DEBUG:cmd:2024-05-22 12:12:35:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:cmd:2024-05-22 12:12:35:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-05-22 12:12:36:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-05-22 12:12:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:12:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: Currently logged in as: 3555028709. Use `wandb login --relogin` to force relogin DEBUG:connectionpool:2024-05-22 12:12:37:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-05-22 12:12:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:pyagent:2024-05-22 12:12:38:Starting sweep agent: entity=None, project=None, count=3 DEBUG:pyagent:2024-05-22 12:12:38:Agent._setup() DEBUG:pyagent:2024-05-22 12:12:38:Agent._register() DEBUG:connectionpool:2024-05-22 12:12:38:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-05-22 12:12:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:pyagent:2024-05-22 12:12:39:agent_id = QWdlbnQ6aDI3bDNhZ24= DEBUG:connectionpool:2024-05-22 12:12:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:pyagent:2024-05-22 12:12:39:Job received: Job(f1csdz3z,{'batch_size': {'value': 40000}, 'dataloader': {'value': 'KGEDataLoader'}, 'delta': {'value': 0.0001}, 'device': {'value': 'cuda:0'}, 'dim': {'value': 400}, 'ent_file': {'value': 'entity2id.txt'}, 'epochs': {'value': 10000}, 'in_path': {'value': '../../benchmarks/FB15K237/'}, 'log_interval': {'value': 100}, 'loss': {'value': 'RGCNLoss'}, 'lr': {'value': 0.4671759876458148}, 'metric': {'value': 'hits@10'}, 'model': {'value': 'RGCN'}, 'neg_ent': {'value': 1}, 'num_layers': {'value': 2}, 'num_workers': {'value': 16}, 'opt_method': {'value': 'sgd'}, 'patience': {'value': 2}, 'prediction': {'value': 'all'}, 'regularization': {'value': 1e-05}, 'rel_file': {'value': 'relation2id.txt'}, 'save_path': {'value': './model.pth'}, 'strategy': {'value': 'RGCNSampling'}, 'test_batch_size': {'value': 30}, 'test_file': {'value': 'test2id.txt'}, 'test_sampler': {'value': 'RGCNTestSampler'}, 'tester': {'value': 'Tester'}, 'train_file': {'value': 'train2id.txt'}, 'train_sampler': {'value': 'RGCNSampler'}, 'trainer': {'value': 'Trainer'}, 'type_constrain': {'value': True}, 'use_early_stopping': {'value': True}, 'use_gpu': {'value': True}, 'use_tqdm': {'value': False}, 'valid_file': {'value': 'valid2id.txt'}, 'valid_interval': {'value': 100}}) DEBUG:pyagent:2024-05-22 12:12:39:Spawning new thread for run f1csdz3z. wandb: Agent Starting Run: f1csdz3z with config: wandb: batch_size: 40000 wandb: dataloader: KGEDataLoader wandb: delta: 0.0001 wandb: device: cuda:0 wandb: dim: 400 wandb: ent_file: entity2id.txt wandb: epochs: 10000 wandb: in_path: ../../benchmarks/FB15K237/ wandb: log_interval: 100 wandb: loss: RGCNLoss wandb: lr: 0.4671759876458148 wandb: metric: hits@10 wandb: model: RGCN wandb: neg_ent: 1 wandb: num_layers: 2 wandb: num_workers: 16 wandb: opt_method: sgd wandb: patience: 2 wandb: prediction: all wandb: regularization: 1e-05 wandb: rel_file: relation2id.txt wandb: save_path: ./model.pth wandb: strategy: RGCNSampling wandb: test_batch_size: 30 wandb: test_file: test2id.txt wandb: test_sampler: RGCNTestSampler wandb: tester: Tester wandb: train_file: train2id.txt wandb: train_sampler: RGCNSampler wandb: trainer: Trainer wandb: type_constrain: True wandb: use_early_stopping: True wandb: use_gpu: True wandb: use_tqdm: False wandb: valid_file: valid2id.txt wandb: valid_interval: 100 DEBUG:cmd:2024-05-22 12:12:39:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:cmd:2024-05-22 12:12:39:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-05-22 12:12:40:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-05-22 12:12:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:12:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:cmd:2024-05-22 12:12:41:Popen(['git', 'cat-file', '--batch-check'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-05-22 12:12:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: wandb version 0.17.0 is available! To upgrade, please run: wandb: $ pip install wandb --upgrade wandb: Tracking run with wandb version 0.16.6 wandb: Run data is saved locally in /home/luyanfeng/my_code/github/pybind11-OpenKE/examples/RGCN/wandb/run-20240522_121241-f1csdz3z wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run snowy-sweep-1 wandb: โญ๏ธ View project at https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: ๐Ÿงน View sweep at https://wandb.ai/3555028709/pybind11-ke-sweeps/sweeps/m256vw5a wandb: ๐Ÿš€ View run at https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/f1csdz3z wandb: Network error (TransientError), entering retry loop. DEBUG:connectionpool:2024-05-22 12:12:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:12:52:[cuda:0] Initialization completed, start model training. data_loader_config: {'batch_size': {'values': [512, 1024, 2048, 4096]}, 'dataloader': {'value': 'KGEDataLoader'}, 'ent_file': {'value': 'entity2id.txt'}, 'in_path': {'value': './'}, 'neg_ent': {'values': [1, 4, 16, 64]}, 'num_workers': {'value': 16}, 'rel_file': {'value': 'relation2id.txt'}, 'test_batch_size': {'value': 30}, 'test_file': {'value': 'test2id.txt'}, 'test_sampler': {'value': 'TradTestSampler'}, 'train_file': {'value': 'train2id.txt'}, 'train_sampler': {'value': 'BernSampler'}, 'type_constrain': {'value': True}, 'valid_file': {'value': 'valid2id.txt'}} kge_config: {'dim': {'values': [200, 300, 400]}, 'model': {'value': 'RGCN'}, 'num_layers': {'value': 2}} loss_config: {'loss': {'value': 'RGCNLoss'}, 'regularization': {'value': 1e-05}} strategy_config: {'strategy': {'value': 'RGCNSampling'}} tester_config: {'device': {'value': 'cuda:0'}, 'prediction': {'value': 'all'}, 'tester': {'value': 'Tester'}, 'use_gpu': {'value': True}, 'use_tqdm': {'value': False}} trainer_config: {'delta': {'value': 0.0001}, 'epochs': {'value': 10000}, 'log_interval': {'value': 100}, 'lr': {'distribution': 'uniform', 'max': 1.0, 'min': 1e-05}, 'metric': {'value': 'hits@10'}, 'opt_method': {'values': ['adam', 'adagrad', 'sgd']}, 'patience': {'value': 2}, 'save_path': {'value': './model.pth'}, 'trainer': {'value': 'Trainer'}, 'use_early_stopping': {'value': True}, 'valid_interval': {'value': 100}} sweep_config: {'method': 'bayes', 'metric': {'goal': 'maximize', 'name': 'val/hits@10'}, 'name': 'RGCN_FB15K237', 'parameters': {'batch_size': {'values': [40000, 50000, 60000, 70000]}, 'dataloader': {'value': 'KGEDataLoader'}, 'delta': {'value': 0.0001}, 'device': {'value': 'cuda:0'}, 'dim': {'values': [200, 300, 400]}, 'ent_file': {'value': 'entity2id.txt'}, 'epochs': {'value': 10000}, 'in_path': {'value': '../../benchmarks/FB15K237/'}, 'log_interval': {'value': 100}, 'loss': {'value': 'RGCNLoss'}, 'lr': {'distribution': 'uniform', 'max': 1.0, 'min': 1e-05}, 'metric': {'value': 'hits@10'}, 'model': {'value': 'RGCN'}, 'neg_ent': {'values': [1, 4, 8, 16]}, 'num_layers': {'value': 2}, 'num_workers': {'value': 16}, 'opt_method': {'values': ['adam', 'adagrad', 'sgd']}, 'patience': {'value': 2}, 'prediction': {'value': 'all'}, 'regularization': {'value': 1e-05}, 'rel_file': {'value': 'relation2id.txt'}, 'save_path': {'value': './model.pth'}, 'strategy': {'value': 'RGCNSampling'}, 'test_batch_size': {'value': 30}, 'test_file': {'value': 'test2id.txt'}, 'test_sampler': {'value': 'RGCNTestSampler'}, 'tester': {'value': 'Tester'}, 'train_file': {'value': 'train2id.txt'}, 'train_sampler': {'value': 'RGCNSampler'}, 'trainer': {'value': 'Trainer'}, 'type_constrain': {'value': True}, 'use_early_stopping': {'value': True}, 'use_gpu': {'value': True}, 'use_tqdm': {'value': False}, 'valid_file': {'value': 'valid2id.txt'}, 'valid_interval': {'value': 100}}} Create sweep with ID: m256vw5a Sweep URL: https://wandb.ai/3555028709/pybind11-ke-sweeps/sweeps/m256vw5a DEBUG:connectionpool:2024-05-22 12:12:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:13:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:13:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:13:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:13:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:13:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:13:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:13:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:13:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:13:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:13:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:13:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:13:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:14:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:14:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:14:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:14:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:14:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:14:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:14:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:14:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:14:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:14:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:14:55:[cuda:0] Epoch 100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 12:14:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:15:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:15:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:15:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:15:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:15:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:15:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:15:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:15:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:15:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:15:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:15:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:16:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:16:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:16:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:16:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:16:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:16:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:16:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:16:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:16:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:16:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:16:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:16:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:17:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:17:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:17:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:17:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:17:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:17:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:17:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:17:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:17:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:17:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:17:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:18:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:18:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:18:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:18:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:18:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:18:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:18:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:18:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:18:42:mr: 583.138 INFO:Trainer:2024-05-22 12:18:42:mrr: 0.141 INFO:Trainer:2024-05-22 12:18:42:hits@1: 0.085 INFO:Trainer:2024-05-22 12:18:42:hits@3: 0.147 INFO:Trainer:2024-05-22 12:18:42:hits@10: 0.247 INFO:Trainer:2024-05-22 12:18:42:mr_type: 265.986 INFO:Trainer:2024-05-22 12:18:42:mrr_type: 0.179 INFO:Trainer:2024-05-22 12:18:42:hits@1_type: 0.116 INFO:Trainer:2024-05-22 12:18:42:hits@3_type: 0.184 INFO:Trainer:2024-05-22 12:18:42:hits@10_type: 0.3 INFO:EarlyStopping:2024-05-22 12:18:42:Validation score improved (-inf --> 0.247000). Saving model ... INFO:Trainer:2024-05-22 12:18:42:[cuda:0] Epoch [ 100/10000] | Batchsize: 40000 | loss: 3.862490 | 1.22855 seconds/epoch DEBUG:connectionpool:2024-05-22 12:18:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:18:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:18:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:18:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:19:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:19:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:19:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:19:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:19:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:19:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:19:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:19:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:19:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:19:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:19:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:20:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:20:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:20:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:20:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:20:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:20:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:20:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:20:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:20:43:[cuda:0] Epoch 200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 12:20:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:20:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:20:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:21:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:21:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:21:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:21:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:21:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:21:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:21:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:21:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:21:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:21:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:21:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:22:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:22:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:22:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:22:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:22:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:22:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:22:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:22:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:22:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:22:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:22:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:22:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:23:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:23:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:23:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:23:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:23:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:23:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:23:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:23:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:23:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:23:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:23:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:24:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:24:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:24:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:24:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:24:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:24:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:24:29:mr: 440.448 INFO:Trainer:2024-05-22 12:24:29:mrr: 0.148 INFO:Trainer:2024-05-22 12:24:29:hits@1: 0.089 INFO:Trainer:2024-05-22 12:24:29:hits@3: 0.154 INFO:Trainer:2024-05-22 12:24:29:hits@10: 0.262 INFO:Trainer:2024-05-22 12:24:29:mr_type: 245.192 INFO:Trainer:2024-05-22 12:24:29:mrr_type: 0.178 INFO:Trainer:2024-05-22 12:24:29:hits@1_type: 0.113 INFO:Trainer:2024-05-22 12:24:29:hits@3_type: 0.183 INFO:Trainer:2024-05-22 12:24:29:hits@10_type: 0.302 INFO:EarlyStopping:2024-05-22 12:24:29:Validation score improved (0.247000 --> 0.262000). Saving model ... INFO:Trainer:2024-05-22 12:24:30:[cuda:0] Epoch [ 200/10000] | Batchsize: 40000 | loss: 3.107359 | 2.35477 seconds/epoch DEBUG:connectionpool:2024-05-22 12:24:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:24:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:24:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:24:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:24:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:24:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:25:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:25:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:25:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:25:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:25:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:25:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:25:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:25:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:25:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:25:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:25:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:26:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:26:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:26:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:26:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:26:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:26:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:26:30:[cuda:0] Epoch 300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 12:26:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:26:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:26:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:26:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:26:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:27:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:27:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:27:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:27:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:27:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:27:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:27:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:27:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:27:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:27:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:27:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:27:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:28:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:28:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:28:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:28:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:28:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:28:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:28:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:28:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:28:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:28:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:28:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:29:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:29:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:29:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:29:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:29:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:29:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:29:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:29:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:29:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:29:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:29:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:29:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:30:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:30:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:30:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:30:17:mr: 395.106 INFO:Trainer:2024-05-22 12:30:17:mrr: 0.153 INFO:Trainer:2024-05-22 12:30:17:hits@1: 0.093 INFO:Trainer:2024-05-22 12:30:17:hits@3: 0.16 INFO:Trainer:2024-05-22 12:30:17:hits@10: 0.27 INFO:Trainer:2024-05-22 12:30:17:mr_type: 232.109 INFO:Trainer:2024-05-22 12:30:17:mrr_type: 0.181 INFO:Trainer:2024-05-22 12:30:17:hits@1_type: 0.116 INFO:Trainer:2024-05-22 12:30:17:hits@3_type: 0.188 INFO:Trainer:2024-05-22 12:30:17:hits@10_type: 0.307 INFO:EarlyStopping:2024-05-22 12:30:17:Validation score improved (0.262000 --> 0.270000). Saving model ... INFO:Trainer:2024-05-22 12:30:17:[cuda:0] Epoch [ 300/10000] | Batchsize: 40000 | loss: 2.777067 | 2.72754 seconds/epoch DEBUG:connectionpool:2024-05-22 12:30:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:30:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:30:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:30:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:30:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:30:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:30:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:30:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:31:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:31:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:31:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:31:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:31:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:31:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:31:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:31:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:31:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:31:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:31:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:32:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:32:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:32:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:32:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:32:18:[cuda:0] Epoch 400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 12:32:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:32:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:32:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:32:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:32:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:32:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:32:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:32:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:33:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:33:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:33:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:33:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:33:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:33:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:33:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:33:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:33:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:33:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:33:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:34:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:34:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:34:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:34:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:34:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:34:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:34:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:34:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:34:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:34:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:34:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:35:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:35:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:35:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:35:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:35:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:35:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:35:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:35:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:35:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:35:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:35:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:35:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:36:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:36:05:mr: 371.15 INFO:Trainer:2024-05-22 12:36:05:mrr: 0.157 INFO:Trainer:2024-05-22 12:36:05:hits@1: 0.094 INFO:Trainer:2024-05-22 12:36:05:hits@3: 0.164 INFO:Trainer:2024-05-22 12:36:05:hits@10: 0.28 INFO:Trainer:2024-05-22 12:36:05:mr_type: 222.886 INFO:Trainer:2024-05-22 12:36:05:mrr_type: 0.184 INFO:Trainer:2024-05-22 12:36:05:hits@1_type: 0.115 INFO:Trainer:2024-05-22 12:36:05:hits@3_type: 0.192 INFO:Trainer:2024-05-22 12:36:05:hits@10_type: 0.316 INFO:EarlyStopping:2024-05-22 12:36:05:Validation score improved (0.270000 --> 0.280000). Saving model ... INFO:Trainer:2024-05-22 12:36:05:[cuda:0] Epoch [ 400/10000] | Batchsize: 40000 | loss: 2.587023 | 2.91610 seconds/epoch DEBUG:connectionpool:2024-05-22 12:36:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:36:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:36:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:36:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:36:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:36:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:36:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:36:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:36:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:36:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:37:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:37:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:37:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:37:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:37:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:37:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:37:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:37:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:37:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:37:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:37:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:38:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:38:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:38:10:[cuda:0] Epoch 500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 12:38:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:38:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:38:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:38:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:38:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:38:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:38:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:38:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:38:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:38:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:39:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:39:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:39:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:39:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:39:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:39:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:39:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:39:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:39:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:39:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:39:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:40:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:40:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:40:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:40:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:40:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:40:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:40:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:40:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:40:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:40:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:40:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:40:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:41:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:41:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:41:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:41:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:41:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:41:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:41:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:41:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:41:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:41:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:41:56:mr: 356.63 INFO:Trainer:2024-05-22 12:41:56:mrr: 0.16 INFO:Trainer:2024-05-22 12:41:56:hits@1: 0.096 INFO:Trainer:2024-05-22 12:41:56:hits@3: 0.166 INFO:Trainer:2024-05-22 12:41:56:hits@10: 0.284 INFO:Trainer:2024-05-22 12:41:56:mr_type: 216.794 INFO:Trainer:2024-05-22 12:41:56:mrr_type: 0.186 INFO:Trainer:2024-05-22 12:41:56:hits@1_type: 0.117 INFO:Trainer:2024-05-22 12:41:56:hits@3_type: 0.193 INFO:Trainer:2024-05-22 12:41:56:hits@10_type: 0.32 INFO:EarlyStopping:2024-05-22 12:41:56:Validation score improved (0.280000 --> 0.284000). Saving model ... INFO:Trainer:2024-05-22 12:41:57:[cuda:0] Epoch [ 500/10000] | Batchsize: 40000 | loss: 2.454941 | 3.03586 seconds/epoch DEBUG:connectionpool:2024-05-22 12:41:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:42:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:42:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:42:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:42:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:42:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:42:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:42:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:42:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:42:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:42:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:42:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:43:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:43:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:43:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:43:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:43:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:43:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:43:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:43:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:43:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:43:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:43:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:43:55:[cuda:0] Epoch 600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 12:43:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:44:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:44:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:44:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:44:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:44:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:44:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:44:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:44:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:44:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:44:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:44:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:45:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:45:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:45:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:45:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:45:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:45:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:45:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:45:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:45:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:45:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:45:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:45:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:46:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:46:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:46:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:46:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:46:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:46:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:46:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:46:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:46:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:46:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:46:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:47:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:47:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:47:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:47:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:47:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:47:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:47:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:47:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:47:42:mr: 345.754 INFO:Trainer:2024-05-22 12:47:42:mrr: 0.162 INFO:Trainer:2024-05-22 12:47:42:hits@1: 0.098 INFO:Trainer:2024-05-22 12:47:42:hits@3: 0.168 INFO:Trainer:2024-05-22 12:47:42:hits@10: 0.287 INFO:Trainer:2024-05-22 12:47:42:mr_type: 210.878 INFO:Trainer:2024-05-22 12:47:42:mrr_type: 0.189 INFO:Trainer:2024-05-22 12:47:42:hits@1_type: 0.12 INFO:Trainer:2024-05-22 12:47:42:hits@3_type: 0.196 INFO:Trainer:2024-05-22 12:47:42:hits@10_type: 0.322 INFO:EarlyStopping:2024-05-22 12:47:42:Validation score improved (0.284000 --> 0.287000). Saving model ... INFO:Trainer:2024-05-22 12:47:42:[cuda:0] Epoch [ 600/10000] | Batchsize: 40000 | loss: 2.345931 | 3.10574 seconds/epoch DEBUG:connectionpool:2024-05-22 12:47:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:47:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:47:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:48:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:48:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:48:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:48:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:48:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:48:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:48:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:48:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:48:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:48:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:48:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:49:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:49:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:49:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:49:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:49:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:49:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:49:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:49:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:49:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:49:45:[cuda:0] Epoch 700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 12:49:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:49:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:49:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:50:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:50:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:50:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:50:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:50:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:50:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:50:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:50:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:50:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:50:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:50:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:51:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:51:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:51:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:51:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:51:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:51:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:51:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:51:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:51:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:51:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:51:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:52:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:52:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:52:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:52:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:52:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:52:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:52:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:52:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:52:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:52:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:52:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:52:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:53:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:53:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:53:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:53:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:53:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:53:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:53:32:mr: 336.229 INFO:Trainer:2024-05-22 12:53:32:mrr: 0.163 INFO:Trainer:2024-05-22 12:53:32:hits@1: 0.097 INFO:Trainer:2024-05-22 12:53:32:hits@3: 0.171 INFO:Trainer:2024-05-22 12:53:32:hits@10: 0.293 INFO:Trainer:2024-05-22 12:53:32:mr_type: 205.578 INFO:Trainer:2024-05-22 12:53:32:mrr_type: 0.189 INFO:Trainer:2024-05-22 12:53:32:hits@1_type: 0.119 INFO:Trainer:2024-05-22 12:53:32:hits@3_type: 0.198 INFO:Trainer:2024-05-22 12:53:32:hits@10_type: 0.329 INFO:EarlyStopping:2024-05-22 12:53:32:Validation score improved (0.287000 --> 0.293000). Saving model ... INFO:Trainer:2024-05-22 12:53:32:[cuda:0] Epoch [ 700/10000] | Batchsize: 40000 | loss: 2.270431 | 3.16185 seconds/epoch DEBUG:connectionpool:2024-05-22 12:53:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:53:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:53:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:53:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:53:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:54:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:54:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:54:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:54:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:54:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:54:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:54:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:54:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:54:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:54:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:54:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:55:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:55:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:55:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:55:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:55:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:55:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:55:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:55:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:55:38:[cuda:0] Epoch 800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 12:55:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:55:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:55:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:55:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:56:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:56:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:56:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:56:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:56:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:56:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:56:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:56:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:56:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:56:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:56:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:57:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:57:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:57:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:57:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:57:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:57:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:57:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:57:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:57:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:57:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:57:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:57:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:58:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:58:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:58:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:58:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:58:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:58:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:58:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:58:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:58:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:58:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:58:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:59:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:59:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:59:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:59:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:59:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 12:59:24:mr: 328.311 INFO:Trainer:2024-05-22 12:59:24:mrr: 0.164 INFO:Trainer:2024-05-22 12:59:24:hits@1: 0.098 INFO:Trainer:2024-05-22 12:59:24:hits@3: 0.171 INFO:Trainer:2024-05-22 12:59:24:hits@10: 0.294 INFO:Trainer:2024-05-22 12:59:24:mr_type: 201.804 INFO:Trainer:2024-05-22 12:59:24:mrr_type: 0.19 INFO:Trainer:2024-05-22 12:59:24:hits@1_type: 0.119 INFO:Trainer:2024-05-22 12:59:24:hits@3_type: 0.199 INFO:Trainer:2024-05-22 12:59:24:hits@10_type: 0.331 INFO:EarlyStopping:2024-05-22 12:59:24:Validation score improved (0.293000 --> 0.294000). Saving model ... INFO:Trainer:2024-05-22 12:59:24:[cuda:0] Epoch [ 800/10000] | Batchsize: 40000 | loss: 2.193019 | 3.20713 seconds/epoch DEBUG:connectionpool:2024-05-22 12:59:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:59:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:59:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:59:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:59:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 12:59:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:00:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:00:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:00:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:00:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:00:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:00:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:00:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:00:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:00:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:00:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:00:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:00:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:01:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:01:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:01:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:01:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:01:25:[cuda:0] Epoch 900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 13:01:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:01:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:01:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:01:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:01:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:01:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:01:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:02:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:02:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:02:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:02:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:02:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:02:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:02:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:02:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:02:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:02:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:02:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:03:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:03:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:03:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:03:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:03:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:03:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:03:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:03:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:03:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:03:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:03:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:03:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:04:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:04:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:04:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:04:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:04:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:04:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:04:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:04:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:04:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:04:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:04:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:05:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:05:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:05:12:mr: 321.585 INFO:Trainer:2024-05-22 13:05:12:mrr: 0.169 INFO:Trainer:2024-05-22 13:05:12:hits@1: 0.103 INFO:Trainer:2024-05-22 13:05:12:hits@3: 0.177 INFO:Trainer:2024-05-22 13:05:12:hits@10: 0.301 INFO:Trainer:2024-05-22 13:05:12:mr_type: 197.713 INFO:Trainer:2024-05-22 13:05:12:mrr_type: 0.196 INFO:Trainer:2024-05-22 13:05:12:hits@1_type: 0.125 INFO:Trainer:2024-05-22 13:05:12:hits@3_type: 0.204 INFO:Trainer:2024-05-22 13:05:12:hits@10_type: 0.337 INFO:EarlyStopping:2024-05-22 13:05:12:Validation score improved (0.294000 --> 0.301000). Saving model ... DEBUG:connectionpool:2024-05-22 13:05:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:05:12:[cuda:0] Epoch [ 900/10000] | Batchsize: 40000 | loss: 2.137370 | 3.23680 seconds/epoch DEBUG:connectionpool:2024-05-22 13:05:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:05:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:05:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:05:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:05:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:05:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:05:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:05:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:05:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:06:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:06:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:06:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:06:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:06:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:06:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:06:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:06:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:06:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:06:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:06:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:07:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:07:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:07:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:07:17:[cuda:0] Epoch 1000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 13:07:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:07:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:07:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:07:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:07:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:07:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:07:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:07:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:08:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:08:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:08:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:08:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:08:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:08:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:08:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:08:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:08:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:08:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:08:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:08:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:09:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:09:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:09:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:09:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:09:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:09:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:09:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:09:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:09:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:09:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:09:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:10:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:10:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:10:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:10:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:10:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:10:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:10:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:10:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:10:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:10:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:10:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:10:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:11:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:11:04:mr: 314.579 INFO:Trainer:2024-05-22 13:11:04:mrr: 0.168 INFO:Trainer:2024-05-22 13:11:04:hits@1: 0.102 INFO:Trainer:2024-05-22 13:11:04:hits@3: 0.176 INFO:Trainer:2024-05-22 13:11:04:hits@10: 0.3 INFO:Trainer:2024-05-22 13:11:04:mr_type: 193.65 INFO:Trainer:2024-05-22 13:11:04:mrr_type: 0.195 INFO:Trainer:2024-05-22 13:11:04:hits@1_type: 0.124 INFO:Trainer:2024-05-22 13:11:04:hits@3_type: 0.203 INFO:Trainer:2024-05-22 13:11:04:hits@10_type: 0.338 INFO:EarlyStopping:2024-05-22 13:11:04:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-05-22 13:11:04:[cuda:0] Epoch [1000/10000] | Batchsize: 40000 | loss: 2.086245 | 3.26511 seconds/epoch DEBUG:connectionpool:2024-05-22 13:11:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:11:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:11:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:11:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:11:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:11:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:11:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:11:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:11:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:11:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:12:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:12:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:12:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:12:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:12:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:12:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:12:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:12:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:12:45:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-22 13:12:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:12:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:12:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:13:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:13:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:13:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:13:15:[cuda:0] Epoch 1100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 13:13:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:13:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:13:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:13:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:13:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:13:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:13:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:13:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:14:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:14:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:14:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:14:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:14:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:14:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:14:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:14:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:14:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:14:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:14:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:14:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:15:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:15:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:15:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:15:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:15:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:15:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:15:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:15:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:15:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:15:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:15:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:16:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:16:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:16:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:16:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:16:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:16:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:16:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:16:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:16:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:16:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:16:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:16:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:17:02:mr: 310.215 INFO:Trainer:2024-05-22 13:17:02:mrr: 0.172 INFO:Trainer:2024-05-22 13:17:02:hits@1: 0.105 INFO:Trainer:2024-05-22 13:17:02:hits@3: 0.179 INFO:Trainer:2024-05-22 13:17:02:hits@10: 0.306 INFO:Trainer:2024-05-22 13:17:02:mr_type: 191.257 INFO:Trainer:2024-05-22 13:17:02:mrr_type: 0.199 INFO:Trainer:2024-05-22 13:17:02:hits@1_type: 0.127 INFO:Trainer:2024-05-22 13:17:02:hits@3_type: 0.207 INFO:Trainer:2024-05-22 13:17:02:hits@10_type: 0.343 INFO:EarlyStopping:2024-05-22 13:17:02:Validation score improved (0.301000 --> 0.306000). Saving model ... INFO:Trainer:2024-05-22 13:17:02:[cuda:0] Epoch [1100/10000] | Batchsize: 40000 | loss: 2.047371 | 3.29370 seconds/epoch DEBUG:connectionpool:2024-05-22 13:17:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:17:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:17:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:17:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:17:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:17:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:17:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:17:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:17:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:17:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:17:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:18:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:18:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:18:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:18:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:18:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:18:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:18:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:18:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:18:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:18:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:18:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:18:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:19:02:[cuda:0] Epoch 1200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 13:19:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:19:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:19:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:19:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:19:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:19:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:19:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:19:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:19:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:19:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:19:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:20:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:20:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:20:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:20:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:20:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:20:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:20:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:20:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:20:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:20:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:20:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:20:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:21:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:21:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:21:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:21:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:21:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:21:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:21:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:21:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:21:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:21:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:21:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:22:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:22:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:22:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:22:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:22:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:22:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:22:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:22:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:22:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:22:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:22:49:mr: 305.988 INFO:Trainer:2024-05-22 13:22:49:mrr: 0.171 INFO:Trainer:2024-05-22 13:22:49:hits@1: 0.104 INFO:Trainer:2024-05-22 13:22:49:hits@3: 0.179 INFO:Trainer:2024-05-22 13:22:49:hits@10: 0.306 INFO:Trainer:2024-05-22 13:22:49:mr_type: 188.441 INFO:Trainer:2024-05-22 13:22:49:mrr_type: 0.198 INFO:Trainer:2024-05-22 13:22:49:hits@1_type: 0.126 INFO:Trainer:2024-05-22 13:22:49:hits@3_type: 0.207 INFO:Trainer:2024-05-22 13:22:49:hits@10_type: 0.344 INFO:EarlyStopping:2024-05-22 13:22:49:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-05-22 13:22:49:[cuda:0] Epoch [1200/10000] | Batchsize: 40000 | loss: 2.006119 | 3.30855 seconds/epoch DEBUG:connectionpool:2024-05-22 13:22:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:22:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:23:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:23:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:23:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:23:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:23:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:23:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:23:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:23:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:23:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:23:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:23:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:24:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:24:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:24:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:24:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:24:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:24:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:24:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:24:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:24:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:24:50:[cuda:0] Epoch 1300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 13:24:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:24:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:25:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:25:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:25:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:25:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:25:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:25:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:25:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:25:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:25:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:25:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:25:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:25:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:26:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:26:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:26:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:26:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:26:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:26:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:26:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:26:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:26:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:26:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:26:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:27:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:27:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:27:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:27:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:27:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:27:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:27:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:27:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:27:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:27:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:27:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:27:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:28:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:28:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:28:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:28:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:28:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:28:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:28:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:28:37:mr: 302.34 INFO:Trainer:2024-05-22 13:28:37:mrr: 0.173 INFO:Trainer:2024-05-22 13:28:37:hits@1: 0.104 INFO:Trainer:2024-05-22 13:28:37:hits@3: 0.181 INFO:Trainer:2024-05-22 13:28:37:hits@10: 0.311 INFO:Trainer:2024-05-22 13:28:37:mr_type: 186.131 INFO:Trainer:2024-05-22 13:28:37:mrr_type: 0.2 INFO:Trainer:2024-05-22 13:28:37:hits@1_type: 0.127 INFO:Trainer:2024-05-22 13:28:37:hits@3_type: 0.209 INFO:Trainer:2024-05-22 13:28:37:hits@10_type: 0.349 INFO:EarlyStopping:2024-05-22 13:28:37:Validation score improved (0.306000 --> 0.311000). Saving model ... INFO:Trainer:2024-05-22 13:28:37:[cuda:0] Epoch [1300/10000] | Batchsize: 40000 | loss: 1.968923 | 3.32139 seconds/epoch DEBUG:connectionpool:2024-05-22 13:28:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:28:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:28:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:28:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:29:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:29:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:29:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:29:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:29:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:29:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:29:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:29:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:29:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:29:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:29:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:29:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:30:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:30:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:30:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:30:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:30:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:30:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:30:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:30:37:[cuda:0] Epoch 1400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 13:30:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:30:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:30:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:30:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:31:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:31:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:31:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:31:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:31:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:31:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:31:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:31:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:31:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:31:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:31:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:32:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:32:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:32:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:32:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:32:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:32:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:32:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:32:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:32:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:32:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:32:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:32:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:33:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:33:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:33:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:33:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:33:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:33:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:33:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:33:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:33:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:33:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:33:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:34:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:34:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:34:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:34:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:34:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:34:24:mr: 298.314 INFO:Trainer:2024-05-22 13:34:24:mrr: 0.174 INFO:Trainer:2024-05-22 13:34:24:hits@1: 0.105 INFO:Trainer:2024-05-22 13:34:24:hits@3: 0.183 INFO:Trainer:2024-05-22 13:34:24:hits@10: 0.311 INFO:Trainer:2024-05-22 13:34:24:mr_type: 183.703 INFO:Trainer:2024-05-22 13:34:24:mrr_type: 0.201 INFO:Trainer:2024-05-22 13:34:24:hits@1_type: 0.128 INFO:Trainer:2024-05-22 13:34:24:hits@3_type: 0.212 INFO:Trainer:2024-05-22 13:34:24:hits@10_type: 0.348 INFO:EarlyStopping:2024-05-22 13:34:24:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-05-22 13:34:24:[cuda:0] Epoch [1400/10000] | Batchsize: 40000 | loss: 1.934786 | 3.33225 seconds/epoch DEBUG:connectionpool:2024-05-22 13:34:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:34:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:34:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:34:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:34:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:34:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:34:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:35:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:35:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:35:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:35:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:35:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:35:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:35:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:35:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:35:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:35:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:35:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:36:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:36:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:36:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:36:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:36:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:36:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:36:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:36:39:[cuda:0] Epoch 1500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 13:36:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:36:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:36:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:36:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:37:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:37:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:37:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:37:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:37:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:37:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:37:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:37:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:37:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:37:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:37:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:37:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:38:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:38:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:38:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:38:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:38:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:38:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:38:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:38:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:38:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:38:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:38:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:39:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:39:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:39:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:39:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:39:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:39:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:39:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:39:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:39:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:39:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:39:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:39:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:40:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:40:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:40:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:40:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:40:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:40:25:mr: 296.071 INFO:Trainer:2024-05-22 13:40:25:mrr: 0.176 INFO:Trainer:2024-05-22 13:40:25:hits@1: 0.107 INFO:Trainer:2024-05-22 13:40:25:hits@3: 0.184 INFO:Trainer:2024-05-22 13:40:25:hits@10: 0.315 INFO:Trainer:2024-05-22 13:40:25:mr_type: 181.956 INFO:Trainer:2024-05-22 13:40:25:mrr_type: 0.204 INFO:Trainer:2024-05-22 13:40:25:hits@1_type: 0.131 INFO:Trainer:2024-05-22 13:40:25:hits@3_type: 0.213 INFO:Trainer:2024-05-22 13:40:25:hits@10_type: 0.353 INFO:EarlyStopping:2024-05-22 13:40:25:Validation score improved (0.311000 --> 0.315000). Saving model ... INFO:Trainer:2024-05-22 13:40:25:[cuda:0] Epoch [1500/10000] | Batchsize: 40000 | loss: 1.913562 | 3.35101 seconds/epoch DEBUG:connectionpool:2024-05-22 13:40:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:40:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:40:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:40:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:40:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:40:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:41:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:41:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:41:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:41:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:41:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:41:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:41:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:41:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:41:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:41:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:41:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:42:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:42:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:42:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:42:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:42:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:42:25:[cuda:0] Epoch 1600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 13:42:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:42:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:42:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:42:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:42:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:42:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:42:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:43:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:43:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:43:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:43:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:43:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:43:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:43:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:43:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:43:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:43:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:43:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:44:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:44:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:44:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:44:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:44:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:44:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:44:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:44:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:44:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:44:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:44:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:44:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:45:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:45:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:45:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:45:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:45:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:45:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:45:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:45:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:45:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:45:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:45:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:46:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:46:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:46:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:46:12:mr: 291.794 INFO:Trainer:2024-05-22 13:46:12:mrr: 0.175 INFO:Trainer:2024-05-22 13:46:12:hits@1: 0.106 INFO:Trainer:2024-05-22 13:46:12:hits@3: 0.182 INFO:Trainer:2024-05-22 13:46:12:hits@10: 0.316 INFO:Trainer:2024-05-22 13:46:12:mr_type: 179.508 INFO:Trainer:2024-05-22 13:46:12:mrr_type: 0.203 INFO:Trainer:2024-05-22 13:46:12:hits@1_type: 0.129 INFO:Trainer:2024-05-22 13:46:12:hits@3_type: 0.212 INFO:Trainer:2024-05-22 13:46:12:hits@10_type: 0.354 INFO:EarlyStopping:2024-05-22 13:46:12:Validation score improved (0.315000 --> 0.316000). Saving model ... INFO:Trainer:2024-05-22 13:46:12:[cuda:0] Epoch [1600/10000] | Batchsize: 40000 | loss: 1.875292 | 3.35803 seconds/epoch DEBUG:connectionpool:2024-05-22 13:46:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:46:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:46:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:46:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:46:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:46:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:46:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:46:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:46:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:47:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:47:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:47:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:47:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:47:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:47:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:47:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:47:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:47:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:47:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:47:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:48:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:48:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:48:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:48:17:[cuda:0] Epoch 1700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 13:48:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:48:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:48:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:48:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:48:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:48:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:48:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:48:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:49:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:49:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:49:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:49:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:49:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:49:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:49:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:49:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:49:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:49:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:49:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:49:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:50:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:50:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:50:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:50:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:50:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:50:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:50:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:50:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:50:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:50:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:50:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:51:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:51:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:51:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:51:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:51:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:51:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:51:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:51:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:51:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:51:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:51:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:51:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:52:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:52:04:mr: 289.228 INFO:Trainer:2024-05-22 13:52:04:mrr: 0.176 INFO:Trainer:2024-05-22 13:52:04:hits@1: 0.107 INFO:Trainer:2024-05-22 13:52:04:hits@3: 0.185 INFO:Trainer:2024-05-22 13:52:04:hits@10: 0.317 INFO:Trainer:2024-05-22 13:52:04:mr_type: 177.654 INFO:Trainer:2024-05-22 13:52:04:mrr_type: 0.204 INFO:Trainer:2024-05-22 13:52:04:hits@1_type: 0.13 INFO:Trainer:2024-05-22 13:52:04:hits@3_type: 0.214 INFO:Trainer:2024-05-22 13:52:04:hits@10_type: 0.355 INFO:EarlyStopping:2024-05-22 13:52:04:Validation score improved (0.316000 --> 0.317000). Saving model ... INFO:Trainer:2024-05-22 13:52:04:[cuda:0] Epoch [1700/10000] | Batchsize: 40000 | loss: 1.866805 | 3.36752 seconds/epoch DEBUG:connectionpool:2024-05-22 13:52:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:52:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:52:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:52:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:52:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:52:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:52:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:52:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:52:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:52:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:53:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:53:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:53:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:53:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:53:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:53:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:53:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:53:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:53:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:53:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:53:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:54:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:54:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:54:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:54:13:[cuda:0] Epoch 1800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 13:54:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:54:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:54:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:54:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:54:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:54:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:54:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:54:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:54:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:55:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:55:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:55:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:55:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:55:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:55:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:55:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:55:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:55:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:55:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:55:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:56:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:56:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:56:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:56:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:56:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:56:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:56:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:56:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:56:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:56:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:56:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:56:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:57:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:57:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:57:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:57:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:57:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:57:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:57:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:57:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:57:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:57:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:57:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:57:59:mr: 287.145 INFO:Trainer:2024-05-22 13:57:59:mrr: 0.177 INFO:Trainer:2024-05-22 13:57:59:hits@1: 0.107 INFO:Trainer:2024-05-22 13:57:59:hits@3: 0.185 INFO:Trainer:2024-05-22 13:57:59:hits@10: 0.319 INFO:Trainer:2024-05-22 13:57:59:mr_type: 176.006 INFO:Trainer:2024-05-22 13:57:59:mrr_type: 0.205 INFO:Trainer:2024-05-22 13:57:59:hits@1_type: 0.13 INFO:Trainer:2024-05-22 13:57:59:hits@3_type: 0.215 INFO:Trainer:2024-05-22 13:57:59:hits@10_type: 0.356 INFO:EarlyStopping:2024-05-22 13:57:59:Validation score improved (0.317000 --> 0.319000). Saving model ... INFO:Trainer:2024-05-22 13:58:00:[cuda:0] Epoch [1800/10000] | Batchsize: 40000 | loss: 1.845613 | 3.37828 seconds/epoch DEBUG:connectionpool:2024-05-22 13:58:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:58:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:58:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:58:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:58:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:58:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:58:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:58:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:58:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:58:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:58:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:59:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:59:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:59:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:59:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:59:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:59:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:59:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:59:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:59:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:59:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 13:59:52:[cuda:0] Epoch 1900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 13:59:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 13:59:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:00:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:00:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:00:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:00:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:00:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:00:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:00:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:00:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:00:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:00:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:00:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:01:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:01:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:01:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:01:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:01:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:01:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:01:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:01:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:01:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:01:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:01:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:01:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:02:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:02:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:02:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:02:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:02:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:02:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:02:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:02:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:02:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:02:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:02:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:03:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:03:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:03:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:03:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:03:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:03:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:03:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:03:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:03:38:mr: 283.505 INFO:Trainer:2024-05-22 14:03:38:mrr: 0.175 INFO:Trainer:2024-05-22 14:03:38:hits@1: 0.106 INFO:Trainer:2024-05-22 14:03:38:hits@3: 0.183 INFO:Trainer:2024-05-22 14:03:38:hits@10: 0.315 INFO:Trainer:2024-05-22 14:03:38:mr_type: 174.021 INFO:Trainer:2024-05-22 14:03:38:mrr_type: 0.204 INFO:Trainer:2024-05-22 14:03:38:hits@1_type: 0.13 INFO:Trainer:2024-05-22 14:03:38:hits@3_type: 0.213 INFO:Trainer:2024-05-22 14:03:38:hits@10_type: 0.352 INFO:EarlyStopping:2024-05-22 14:03:38:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-05-22 14:03:38:[cuda:0] Epoch [1900/10000] | Batchsize: 40000 | loss: 1.813021 | 3.37899 seconds/epoch DEBUG:connectionpool:2024-05-22 14:03:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:03:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:03:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:03:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:04:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:04:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:04:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:04:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:04:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:04:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:04:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:04:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:04:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:04:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:04:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:05:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:05:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:05:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:05:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:05:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:05:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:05:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:05:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:05:46:[cuda:0] Epoch 2000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 14:05:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:05:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:05:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:06:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:06:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:06:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:06:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:06:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:06:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:06:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:06:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:06:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:06:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:06:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:07:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:07:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:07:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:07:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:07:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:07:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:07:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:07:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:07:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:07:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:07:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:07:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:08:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:08:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:08:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:08:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:08:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:08:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:08:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:08:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:08:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:08:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:08:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:09:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:09:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:09:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:09:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:09:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:09:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:09:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:09:33:mr: 281.741 INFO:Trainer:2024-05-22 14:09:33:mrr: 0.178 INFO:Trainer:2024-05-22 14:09:33:hits@1: 0.108 INFO:Trainer:2024-05-22 14:09:33:hits@3: 0.185 INFO:Trainer:2024-05-22 14:09:33:hits@10: 0.32 INFO:Trainer:2024-05-22 14:09:33:mr_type: 172.351 INFO:Trainer:2024-05-22 14:09:33:mrr_type: 0.206 INFO:Trainer:2024-05-22 14:09:33:hits@1_type: 0.131 INFO:Trainer:2024-05-22 14:09:33:hits@3_type: 0.216 INFO:Trainer:2024-05-22 14:09:33:hits@10_type: 0.359 INFO:EarlyStopping:2024-05-22 14:09:33:Validation score improved (0.319000 --> 0.320000). Saving model ... INFO:Trainer:2024-05-22 14:09:33:[cuda:0] Epoch [2000/10000] | Batchsize: 40000 | loss: 1.797380 | 3.38708 seconds/epoch DEBUG:connectionpool:2024-05-22 14:09:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:09:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:09:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:09:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:09:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:10:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:10:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:10:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:10:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:10:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:10:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:10:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:10:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:10:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:10:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:10:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:11:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:11:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:11:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:11:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:11:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:11:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:11:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:11:36:[cuda:0] Epoch 2100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 14:11:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:11:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:11:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:11:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:12:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:12:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:12:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:12:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:12:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:12:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:12:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:12:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:12:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:12:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:12:52:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-22 14:12:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:12:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:13:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:13:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:13:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:13:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:13:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:13:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:13:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:13:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:13:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:13:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:13:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:14:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:14:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:14:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:14:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:14:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:14:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:14:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:14:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:14:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:14:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:14:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:14:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:15:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:15:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:15:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:15:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:15:23:mr: 278.237 INFO:Trainer:2024-05-22 14:15:23:mrr: 0.179 INFO:Trainer:2024-05-22 14:15:23:hits@1: 0.109 INFO:Trainer:2024-05-22 14:15:23:hits@3: 0.187 INFO:Trainer:2024-05-22 14:15:23:hits@10: 0.321 INFO:Trainer:2024-05-22 14:15:23:mr_type: 170.203 INFO:Trainer:2024-05-22 14:15:23:mrr_type: 0.207 INFO:Trainer:2024-05-22 14:15:23:hits@1_type: 0.133 INFO:Trainer:2024-05-22 14:15:23:hits@3_type: 0.216 INFO:Trainer:2024-05-22 14:15:23:hits@10_type: 0.359 INFO:EarlyStopping:2024-05-22 14:15:23:Validation score improved (0.320000 --> 0.321000). Saving model ... INFO:Trainer:2024-05-22 14:15:23:[cuda:0] Epoch [2100/10000] | Batchsize: 40000 | loss: 1.783727 | 3.39248 seconds/epoch DEBUG:connectionpool:2024-05-22 14:15:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:15:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:15:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:15:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:15:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:15:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:15:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:16:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:16:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:16:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:16:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:16:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:16:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:16:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:16:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:16:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:16:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:16:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:17:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:17:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:17:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:17:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:17:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:17:24:[cuda:0] Epoch 2200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 14:17:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:17:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:17:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:17:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:17:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:17:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:17:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:18:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:18:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:18:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:18:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:18:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:18:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:18:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:18:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:18:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:18:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:18:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:19:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:19:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:19:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:19:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:19:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:19:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:19:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:19:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:19:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:19:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:19:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:19:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:20:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:20:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:20:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:20:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:20:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:20:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:20:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:20:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:20:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:20:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:20:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:21:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:21:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:21:10:mr: 276.213 INFO:Trainer:2024-05-22 14:21:10:mrr: 0.178 INFO:Trainer:2024-05-22 14:21:10:hits@1: 0.106 INFO:Trainer:2024-05-22 14:21:10:hits@3: 0.187 INFO:Trainer:2024-05-22 14:21:10:hits@10: 0.321 INFO:Trainer:2024-05-22 14:21:10:mr_type: 168.749 INFO:Trainer:2024-05-22 14:21:10:mrr_type: 0.206 INFO:Trainer:2024-05-22 14:21:10:hits@1_type: 0.13 INFO:Trainer:2024-05-22 14:21:10:hits@3_type: 0.217 INFO:Trainer:2024-05-22 14:21:10:hits@10_type: 0.359 INFO:EarlyStopping:2024-05-22 14:21:10:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-05-22 14:21:10:[cuda:0] Epoch [2200/10000] | Batchsize: 40000 | loss: 1.768917 | 3.39626 seconds/epoch DEBUG:connectionpool:2024-05-22 14:21:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:21:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:21:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:21:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:21:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:21:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:21:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:21:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:21:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:21:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:22:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:22:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:22:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:22:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:22:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:22:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:22:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:22:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:22:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:22:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:22:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:23:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:23:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:23:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:23:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:23:20:[cuda:0] Epoch 2300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 14:23:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:23:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:23:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:23:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:23:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:23:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:23:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:24:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:24:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:24:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:24:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:24:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:24:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:24:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:24:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:24:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:24:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:24:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:24:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:25:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:25:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:25:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:25:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:25:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:25:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:25:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:25:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:25:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:25:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:25:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:26:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:26:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:26:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:26:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:26:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:26:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:26:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:26:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:26:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:26:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:26:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:26:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:27:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:27:06:mr: 274.47 INFO:Trainer:2024-05-22 14:27:06:mrr: 0.18 INFO:Trainer:2024-05-22 14:27:06:hits@1: 0.109 INFO:Trainer:2024-05-22 14:27:06:hits@3: 0.189 INFO:Trainer:2024-05-22 14:27:06:hits@10: 0.323 INFO:Trainer:2024-05-22 14:27:06:mr_type: 167.278 INFO:Trainer:2024-05-22 14:27:06:mrr_type: 0.208 INFO:Trainer:2024-05-22 14:27:06:hits@1_type: 0.133 INFO:Trainer:2024-05-22 14:27:06:hits@3_type: 0.219 INFO:Trainer:2024-05-22 14:27:06:hits@10_type: 0.361 INFO:EarlyStopping:2024-05-22 14:27:06:Validation score improved (0.321000 --> 0.323000). Saving model ... INFO:Trainer:2024-05-22 14:27:07:[cuda:0] Epoch [2300/10000] | Batchsize: 40000 | loss: 1.740409 | 3.40357 seconds/epoch DEBUG:connectionpool:2024-05-22 14:27:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:27:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:27:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:27:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:27:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:27:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:27:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:27:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:27:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:27:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:28:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:28:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:28:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:28:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:28:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:28:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:28:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:28:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:28:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:28:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:28:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:28:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:29:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:29:07:[cuda:0] Epoch 2400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 14:29:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:29:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:29:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:29:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:29:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:29:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:29:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:29:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:29:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:29:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:30:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:30:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:30:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:30:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:30:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:30:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:30:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:30:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:30:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:30:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:30:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:30:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:31:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:31:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:31:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:31:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:31:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:31:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:31:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:31:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:31:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:31:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:31:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:32:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:32:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:32:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:32:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:32:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:32:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:32:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:32:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:32:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:32:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:32:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:32:53:mr: 272.379 INFO:Trainer:2024-05-22 14:32:53:mrr: 0.182 INFO:Trainer:2024-05-22 14:32:53:hits@1: 0.112 INFO:Trainer:2024-05-22 14:32:53:hits@3: 0.191 INFO:Trainer:2024-05-22 14:32:53:hits@10: 0.326 INFO:Trainer:2024-05-22 14:32:53:mr_type: 165.598 INFO:Trainer:2024-05-22 14:32:53:mrr_type: 0.211 INFO:Trainer:2024-05-22 14:32:53:hits@1_type: 0.135 INFO:Trainer:2024-05-22 14:32:53:hits@3_type: 0.221 INFO:Trainer:2024-05-22 14:32:53:hits@10_type: 0.364 INFO:EarlyStopping:2024-05-22 14:32:53:Validation score improved (0.323000 --> 0.326000). Saving model ... INFO:Trainer:2024-05-22 14:32:53:[cuda:0] Epoch [2400/10000] | Batchsize: 40000 | loss: 1.737474 | 3.40621 seconds/epoch DEBUG:connectionpool:2024-05-22 14:32:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:33:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:33:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:33:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:33:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:33:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:33:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:33:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:33:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:33:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:33:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:33:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:34:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:34:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:34:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:34:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:34:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:34:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:34:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:34:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:34:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:34:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:34:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:35:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:35:05:[cuda:0] Epoch 2500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 14:35:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:35:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:35:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:35:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:35:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:35:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:35:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:35:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:35:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:35:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:35:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:36:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:36:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:36:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:36:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:36:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:36:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:36:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:36:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:36:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:36:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:36:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:37:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:37:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:37:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:37:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:37:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:37:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:37:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:37:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:37:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:37:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:37:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:37:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:38:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:38:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:38:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:38:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:38:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:38:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:38:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:38:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:38:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:38:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:38:52:mr: 270.058 INFO:Trainer:2024-05-22 14:38:52:mrr: 0.183 INFO:Trainer:2024-05-22 14:38:52:hits@1: 0.112 INFO:Trainer:2024-05-22 14:38:52:hits@3: 0.193 INFO:Trainer:2024-05-22 14:38:52:hits@10: 0.328 INFO:Trainer:2024-05-22 14:38:52:mr_type: 164.434 INFO:Trainer:2024-05-22 14:38:52:mrr_type: 0.212 INFO:Trainer:2024-05-22 14:38:52:hits@1_type: 0.136 INFO:Trainer:2024-05-22 14:38:52:hits@3_type: 0.223 INFO:Trainer:2024-05-22 14:38:52:hits@10_type: 0.366 INFO:EarlyStopping:2024-05-22 14:38:52:Validation score improved (0.326000 --> 0.328000). Saving model ... INFO:Trainer:2024-05-22 14:38:52:[cuda:0] Epoch [2500/10000] | Batchsize: 40000 | loss: 1.725205 | 3.41307 seconds/epoch DEBUG:connectionpool:2024-05-22 14:38:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:39:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:39:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:39:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:39:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:39:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:39:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:39:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:39:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:39:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:39:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:39:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:40:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:40:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:40:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:40:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:40:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:40:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:40:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:40:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:40:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:40:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:40:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:40:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:41:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:41:07:[cuda:0] Epoch 2600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 14:41:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:41:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:41:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:41:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:41:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:41:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:41:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:41:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:41:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:41:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:42:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:42:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:42:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:42:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:42:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:42:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:42:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:42:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:42:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:42:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:42:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:42:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:43:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:43:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:43:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:43:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:43:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:43:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:43:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:43:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:43:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:43:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:43:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:44:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:44:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:44:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:44:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:44:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:44:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:44:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:44:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:44:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:44:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:44:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:44:54:mr: 268.528 INFO:Trainer:2024-05-22 14:44:54:mrr: 0.183 INFO:Trainer:2024-05-22 14:44:54:hits@1: 0.111 INFO:Trainer:2024-05-22 14:44:54:hits@3: 0.194 INFO:Trainer:2024-05-22 14:44:54:hits@10: 0.328 INFO:Trainer:2024-05-22 14:44:54:mr_type: 163.829 INFO:Trainer:2024-05-22 14:44:54:mrr_type: 0.212 INFO:Trainer:2024-05-22 14:44:54:hits@1_type: 0.135 INFO:Trainer:2024-05-22 14:44:54:hits@3_type: 0.224 INFO:Trainer:2024-05-22 14:44:54:hits@10_type: 0.366 INFO:EarlyStopping:2024-05-22 14:44:54:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-05-22 14:44:54:[cuda:0] Epoch [2600/10000] | Batchsize: 40000 | loss: 1.708928 | 3.42120 seconds/epoch DEBUG:connectionpool:2024-05-22 14:44:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:45:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:45:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:45:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:45:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:45:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:45:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:45:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:45:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:45:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:45:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:45:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:46:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:46:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:46:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:46:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:46:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:46:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:46:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:46:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:46:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:46:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:46:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:47:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:47:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:47:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:47:14:[cuda:0] Epoch 2700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 14:47:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:47:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:47:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:47:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:47:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:47:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:47:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:47:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:48:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:48:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:48:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:48:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:48:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:48:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:48:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:48:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:48:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:48:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:48:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:48:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:49:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:49:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:49:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:49:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:49:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:49:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:49:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:49:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:49:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:49:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:49:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:50:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:50:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:50:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:50:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:50:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:50:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:50:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:50:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:50:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:50:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:50:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:50:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:51:01:mr: 267.017 INFO:Trainer:2024-05-22 14:51:01:mrr: 0.185 INFO:Trainer:2024-05-22 14:51:01:hits@1: 0.114 INFO:Trainer:2024-05-22 14:51:01:hits@3: 0.194 INFO:Trainer:2024-05-22 14:51:01:hits@10: 0.328 INFO:Trainer:2024-05-22 14:51:01:mr_type: 162.38 INFO:Trainer:2024-05-22 14:51:01:mrr_type: 0.213 INFO:Trainer:2024-05-22 14:51:01:hits@1_type: 0.137 INFO:Trainer:2024-05-22 14:51:01:hits@3_type: 0.224 INFO:Trainer:2024-05-22 14:51:01:hits@10_type: 0.367 INFO:EarlyStopping:2024-05-22 14:51:01:EarlyStopping counter: 2 / 2 INFO:Trainer:2024-05-22 14:51:01:[cuda:0] Send an early stopping signal INFO:Trainer:2024-05-22 14:51:01:[cuda:0] The model training is completed, taking a total of 9261.62475 seconds. INFO:Trainer:2024-05-22 14:51:01:[cuda:0] Model saved at ./model.pth. INFO:Trainer:2024-05-22 14:51:01:[cuda:0] The model starts evaluating in the test set. DEBUG:connectionpool:2024-05-22 14:51:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:51:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:51:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:51:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:51:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:51:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:51:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:51:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:51:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:51:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:51:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:52:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:52:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:52:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:52:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:52:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:52:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:52:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:52:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:52:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:52:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:52:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:52:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:53:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:53:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:53:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:53:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:53:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:53:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:53:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:53:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:53:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:53:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:53:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:54:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:54:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:54:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:54:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:54:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:54:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:54:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:54:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:54:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:54:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:54:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:54:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:55:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:55:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:55:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:55:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:55:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:55:26:mr: 276.378 INFO:Trainer:2024-05-22 14:55:26:mrr: 0.18 INFO:Trainer:2024-05-22 14:55:26:hits@1: 0.109 INFO:Trainer:2024-05-22 14:55:26:hits@3: 0.189 INFO:Trainer:2024-05-22 14:55:26:hits@10: 0.326 INFO:Trainer:2024-05-22 14:55:26:mr_type: 165.034 INFO:Trainer:2024-05-22 14:55:26:mrr_type: 0.208 INFO:Trainer:2024-05-22 14:55:26:hits@1_type: 0.132 INFO:Trainer:2024-05-22 14:55:26:hits@3_type: 0.218 INFO:Trainer:2024-05-22 14:55:26:hits@10_type: 0.362 wandb: - 0.014 MB of 0.014 MB uploaded wandb: \ 0.014 MB of 0.014 MB uploaded DEBUG:connectionpool:2024-05-22 14:55:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: | 0.014 MB of 0.232 MB uploaded wandb: / 0.018 MB of 0.232 MB uploaded wandb: - 0.127 MB of 0.232 MB uploaded wandb: \ 0.232 MB of 0.232 MB uploaded wandb: | 0.232 MB of 0.232 MB uploaded wandb: / 0.232 MB of 0.232 MB uploaded wandb: DEBUG:connectionpool:2024-05-22 14:55:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: wandb: Run history: wandb: duration โ– wandb: test/hits@1 โ– wandb: test/hits@10 โ– wandb: test/hits@10_type โ– wandb: test/hits@1_type โ– wandb: test/hits@3 โ– wandb: test/hits@3_type โ– wandb: test/mr โ– wandb: test/mr_type โ– wandb: test/mrr โ– wandb: test/mrr_type โ– wandb: train/epoch โ–โ–โ–‚โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆ wandb: train/train_loss โ–ˆโ–†โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ– wandb: val/epoch โ–โ–โ–‚โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–ƒโ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆ wandb: val/hits@1 โ–โ–‚โ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–†โ–‡โ–ˆโ–ˆโ–‡โ–ˆ wandb: val/hits@10 โ–โ–‚โ–ƒโ–„โ–„โ–„โ–…โ–…โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@10_type โ–โ–โ–‚โ–ƒโ–ƒโ–ƒโ–„โ–„โ–…โ–…โ–…โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–†โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@1_type โ–‚โ–โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–…โ–„โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–†โ–‡โ–‡โ–ˆโ–‡โ–ˆ wandb: val/hits@3 โ–โ–‚โ–ƒโ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–‡โ–†โ–‡โ–‡โ–†โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@3_type โ–โ–โ–‚โ–ƒโ–ƒโ–ƒโ–„โ–„โ–…โ–„โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆ wandb: val/mr โ–ˆโ–…โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ– wandb: val/mr_type โ–ˆโ–‡โ–†โ–…โ–…โ–„โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ– wandb: val/mrr โ–โ–‚โ–ƒโ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–‡โ–†โ–‡โ–‡โ–†โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/mrr_type โ–โ–โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–ƒโ–…โ–„โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆ wandb: wandb: Run summary: wandb: duration 9261.62475 wandb: test/hits@1 0.109 wandb: test/hits@10 0.326 wandb: test/hits@10_type 0.362 wandb: test/hits@1_type 0.132 wandb: test/hits@3 0.189 wandb: test/hits@3_type 0.218 wandb: test/mr 276.378 wandb: test/mr_type 165.034 wandb: test/mrr 0.18 wandb: test/mrr_type 0.208 wandb: train/epoch 2600 wandb: train/train_loss 1.70893 wandb: val/epoch 2699 wandb: val/hits@1 0.114 wandb: val/hits@10 0.328 wandb: val/hits@10_type 0.367 wandb: val/hits@1_type 0.137 wandb: val/hits@3 0.194 wandb: val/hits@3_type 0.224 wandb: val/mr 267.017 wandb: val/mr_type 162.38 wandb: val/mrr 0.185 wandb: val/mrr_type 0.213 wandb: wandb: ๐Ÿš€ View run snowy-sweep-1 at: https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/f1csdz3z wandb: โญ๏ธ View project at: https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) wandb: Find logs at: ./wandb/run-20240522_121241-f1csdz3z/logs DEBUG:connectionpool:2024-05-22 14:55:34:Starting new HTTPS connection (1): o151352.ingest.sentry.io:443 DEBUG:connectionpool:2024-05-22 14:55:35:https://o151352.ingest.sentry.io:443 "POST /api/4504800232407040/envelope/ HTTP/1.1" 200 0 DEBUG:connectionpool:2024-05-22 14:55:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: Agent Starting Run: 1bewf55d with config: wandb: batch_size: 60000 wandb: dataloader: KGEDataLoader wandb: delta: 0.0001 wandb: device: cuda:0 wandb: dim: 200 wandb: ent_file: entity2id.txt wandb: epochs: 10000 wandb: in_path: ../../benchmarks/FB15K237/ wandb: log_interval: 100 wandb: loss: RGCNLoss wandb: lr: 0.6097411966544464 wandb: metric: hits@10 wandb: model: RGCN wandb: neg_ent: 1 wandb: num_layers: 2 wandb: num_workers: 16 wandb: opt_method: adagrad wandb: patience: 2 wandb: prediction: all wandb: regularization: 1e-05 wandb: rel_file: relation2id.txt wandb: save_path: ./model.pth wandb: strategy: RGCNSampling wandb: test_batch_size: 30 wandb: test_file: test2id.txt wandb: test_sampler: RGCNTestSampler wandb: tester: Tester wandb: train_file: train2id.txt wandb: train_sampler: RGCNSampler wandb: trainer: Trainer wandb: type_constrain: True wandb: use_early_stopping: True wandb: use_gpu: True wandb: use_tqdm: False wandb: valid_file: valid2id.txt wandb: valid_interval: 100 DEBUG:cmd:2024-05-22 14:55:41:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:cmd:2024-05-22 14:55:41:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-05-22 14:55:41:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-05-22 14:55:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:55:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:cmd:2024-05-22 14:55:42:Popen(['git', 'cat-file', '--batch-check'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-05-22 14:55:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: wandb version 0.17.0 is available! To upgrade, please run: wandb: $ pip install wandb --upgrade wandb: Tracking run with wandb version 0.16.6 wandb: Run data is saved locally in /home/luyanfeng/my_code/github/pybind11-OpenKE/examples/RGCN/wandb/run-20240522_145542-1bewf55d wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run whole-sweep-2 wandb: โญ๏ธ View project at https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: ๐Ÿงน View sweep at https://wandb.ai/3555028709/pybind11-ke-sweeps/sweeps/m256vw5a wandb: ๐Ÿš€ View run at https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/1bewf55d DEBUG:connectionpool:2024-05-22 14:55:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:55:50:[cuda:0] Initialization completed, start model training. DEBUG:connectionpool:2024-05-22 14:55:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:56:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:56:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:56:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:56:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:56:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:56:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:56:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:56:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:56:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:56:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:56:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:57:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:57:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:57:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:57:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:57:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:57:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:57:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:57:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:57:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:57:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:57:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:58:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:58:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:58:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:58:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:58:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:58:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 14:58:27:[cuda:0] Epoch 100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 14:58:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:58:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:58:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:58:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:58:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:58:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:59:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:59:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:59:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:59:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:59:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:59:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:59:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:59:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:59:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:59:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 14:59:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:00:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:00:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:00:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:00:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:00:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:00:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:00:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:00:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:00:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:00:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:00:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:00:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:01:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:01:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:01:09:mr: 4762.789 INFO:Trainer:2024-05-22 15:01:09:mrr: 0.001 INFO:Trainer:2024-05-22 15:01:09:hits@1: 0.0 INFO:Trainer:2024-05-22 15:01:09:hits@3: 0.0 INFO:Trainer:2024-05-22 15:01:09:hits@10: 0.0 INFO:Trainer:2024-05-22 15:01:09:mr_type: 349.235 INFO:Trainer:2024-05-22 15:01:09:mrr_type: 0.133 INFO:Trainer:2024-05-22 15:01:09:hits@1_type: 0.081 INFO:Trainer:2024-05-22 15:01:09:hits@3_type: 0.133 INFO:Trainer:2024-05-22 15:01:09:hits@10_type: 0.237 INFO:EarlyStopping:2024-05-22 15:01:09:Validation score improved (-inf --> 0.000000). Saving model ... INFO:Trainer:2024-05-22 15:01:09:[cuda:0] Epoch [ 100/10000] | Batchsize: 60000 | loss: 6.325680 | 1.56928 seconds/epoch DEBUG:connectionpool:2024-05-22 15:01:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:01:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:01:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:01:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:01:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:01:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:01:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:01:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:01:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:02:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:02:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:02:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:02:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:02:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:02:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:02:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:02:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:02:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:02:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:02:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:03:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:03:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:03:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:03:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:03:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:03:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:03:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:03:35:[cuda:0] Epoch 200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 15:03:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:03:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:03:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:03:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:04:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:04:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:04:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:04:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:04:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:04:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:04:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:04:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:04:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:04:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:04:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:04:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:05:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:05:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:05:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:05:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:05:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:05:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:05:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:05:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:05:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:05:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:05:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:06:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:06:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:06:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:06:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:06:18:mr: 4625.027 INFO:Trainer:2024-05-22 15:06:18:mrr: 0.002 INFO:Trainer:2024-05-22 15:06:18:hits@1: 0.0 INFO:Trainer:2024-05-22 15:06:18:hits@3: 0.0 INFO:Trainer:2024-05-22 15:06:18:hits@10: 0.0 INFO:Trainer:2024-05-22 15:06:18:mr_type: 348.026 INFO:Trainer:2024-05-22 15:06:18:mrr_type: 0.134 INFO:Trainer:2024-05-22 15:06:18:hits@1_type: 0.08 INFO:Trainer:2024-05-22 15:06:18:hits@3_type: 0.131 INFO:Trainer:2024-05-22 15:06:18:hits@10_type: 0.24 INFO:EarlyStopping:2024-05-22 15:06:18:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-05-22 15:06:18:[cuda:0] Epoch [ 200/10000] | Batchsize: 60000 | loss: 6.121858 | 2.32576 seconds/epoch DEBUG:connectionpool:2024-05-22 15:06:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:06:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:06:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:06:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:06:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:06:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:06:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:06:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:07:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:07:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:07:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:07:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:07:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:07:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:07:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:07:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:07:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:07:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:07:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:08:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:08:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:08:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:08:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:08:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:08:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:08:32:[cuda:0] Epoch 300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 15:08:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:08:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:08:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:08:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:08:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:09:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:09:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:09:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:09:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:09:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:09:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:09:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:09:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:09:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:09:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:09:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:09:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:10:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:10:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:10:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:10:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:10:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:10:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:10:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:10:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:10:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:10:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:10:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:11:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:11:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:11:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:11:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:11:19:mr: 4545.641 INFO:Trainer:2024-05-22 15:11:19:mrr: 0.002 INFO:Trainer:2024-05-22 15:11:19:hits@1: 0.0 INFO:Trainer:2024-05-22 15:11:19:hits@3: 0.0 INFO:Trainer:2024-05-22 15:11:19:hits@10: 0.0 INFO:Trainer:2024-05-22 15:11:19:mr_type: 347.804 INFO:Trainer:2024-05-22 15:11:19:mrr_type: 0.138 INFO:Trainer:2024-05-22 15:11:19:hits@1_type: 0.086 INFO:Trainer:2024-05-22 15:11:19:hits@3_type: 0.138 INFO:Trainer:2024-05-22 15:11:19:hits@10_type: 0.234 INFO:EarlyStopping:2024-05-22 15:11:19:EarlyStopping counter: 2 / 2 INFO:Trainer:2024-05-22 15:11:19:[cuda:0] Send an early stopping signal INFO:Trainer:2024-05-22 15:11:19:[cuda:0] The model training is completed, taking a total of 762.09510 seconds. INFO:Trainer:2024-05-22 15:11:19:[cuda:0] Model saved at ./model.pth. INFO:Trainer:2024-05-22 15:11:19:[cuda:0] The model starts evaluating in the test set. DEBUG:connectionpool:2024-05-22 15:11:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:11:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:11:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:11:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:11:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:11:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:11:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:11:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:12:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:12:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:12:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:12:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:12:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:12:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:12:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:12:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:12:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:12:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:12:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:13:00:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-22 15:13:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:13:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:13:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:13:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:13:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:13:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:13:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:13:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:13:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:13:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:13:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:13:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:14:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:14:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:14:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:14:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:14:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:14:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:14:34:mr: 4561.281 INFO:Trainer:2024-05-22 15:14:34:mrr: 0.002 INFO:Trainer:2024-05-22 15:14:34:hits@1: 0.0 INFO:Trainer:2024-05-22 15:14:34:hits@3: 0.0 INFO:Trainer:2024-05-22 15:14:34:hits@10: 0.0 INFO:Trainer:2024-05-22 15:14:34:mr_type: 353.566 INFO:Trainer:2024-05-22 15:14:34:mrr_type: 0.135 INFO:Trainer:2024-05-22 15:14:34:hits@1_type: 0.083 INFO:Trainer:2024-05-22 15:14:34:hits@3_type: 0.136 INFO:Trainer:2024-05-22 15:14:34:hits@10_type: 0.229 DEBUG:connectionpool:2024-05-22 15:14:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: - 0.014 MB of 0.014 MB uploaded wandb: \ 0.014 MB of 0.057 MB uploaded wandb: | 0.034 MB of 0.057 MB uploaded wandb: / 0.034 MB of 0.057 MB uploaded wandb: - 0.034 MB of 0.057 MB uploaded wandb: \ 0.057 MB of 0.057 MB uploaded wandb: DEBUG:connectionpool:2024-05-22 15:14:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: wandb: Run history: wandb: duration โ– wandb: test/hits@1 โ– wandb: test/hits@10 โ– wandb: test/hits@10_type โ– wandb: test/hits@1_type โ– wandb: test/hits@3 โ– wandb: test/hits@3_type โ– wandb: test/mr โ– wandb: test/mr_type โ– wandb: test/mrr โ– wandb: test/mrr_type โ– wandb: train/epoch โ–โ–ˆ wandb: train/train_loss โ–ˆโ– wandb: val/epoch โ–โ–…โ–ˆ wandb: val/hits@1 โ–โ–โ– wandb: val/hits@10 โ–โ–โ– wandb: val/hits@10_type โ–…โ–ˆโ– wandb: val/hits@1_type โ–‚โ–โ–ˆ wandb: val/hits@3 โ–โ–โ– wandb: val/hits@3_type โ–ƒโ–โ–ˆ wandb: val/mr โ–ˆโ–„โ– wandb: val/mr_type โ–ˆโ–‚โ– wandb: val/mrr โ–โ–ˆโ–ˆ wandb: val/mrr_type โ–โ–‚โ–ˆ wandb: wandb: Run summary: wandb: duration 762.0951 wandb: test/hits@1 0.0 wandb: test/hits@10 0.0 wandb: test/hits@10_type 0.229 wandb: test/hits@1_type 0.083 wandb: test/hits@3 0.0 wandb: test/hits@3_type 0.136 wandb: test/mr 4561.281 wandb: test/mr_type 353.566 wandb: test/mrr 0.002 wandb: test/mrr_type 0.135 wandb: train/epoch 200 wandb: train/train_loss 6.12186 wandb: val/epoch 299 wandb: val/hits@1 0.0 wandb: val/hits@10 0.0 wandb: val/hits@10_type 0.234 wandb: val/hits@1_type 0.086 wandb: val/hits@3 0.0 wandb: val/hits@3_type 0.138 wandb: val/mr 4545.641 wandb: val/mr_type 347.804 wandb: val/mrr 0.002 wandb: val/mrr_type 0.138 wandb: wandb: ๐Ÿš€ View run whole-sweep-2 at: https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/1bewf55d wandb: โญ๏ธ View project at: https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) wandb: Find logs at: ./wandb/run-20240522_145542-1bewf55d/logs DEBUG:connectionpool:2024-05-22 15:14:42:Resetting dropped connection: o151352.ingest.sentry.io DEBUG:connectionpool:2024-05-22 15:14:42:https://o151352.ingest.sentry.io:443 "POST /api/4504800232407040/envelope/ HTTP/1.1" 200 0 DEBUG:connectionpool:2024-05-22 15:14:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: Agent Starting Run: 7jqvkcn1 with config: wandb: batch_size: 70000 wandb: dataloader: KGEDataLoader wandb: delta: 0.0001 wandb: device: cuda:0 wandb: dim: 200 wandb: ent_file: entity2id.txt wandb: epochs: 10000 wandb: in_path: ../../benchmarks/FB15K237/ wandb: log_interval: 100 wandb: loss: RGCNLoss wandb: lr: 0.17176484173650544 wandb: metric: hits@10 wandb: model: RGCN wandb: neg_ent: 8 wandb: num_layers: 2 wandb: num_workers: 16 wandb: opt_method: sgd wandb: patience: 2 wandb: prediction: all wandb: regularization: 1e-05 wandb: rel_file: relation2id.txt wandb: save_path: ./model.pth wandb: strategy: RGCNSampling wandb: test_batch_size: 30 wandb: test_file: test2id.txt wandb: test_sampler: RGCNTestSampler wandb: tester: Tester wandb: train_file: train2id.txt wandb: train_sampler: RGCNSampler wandb: trainer: Trainer wandb: type_constrain: True wandb: use_early_stopping: True wandb: use_gpu: True wandb: use_tqdm: False wandb: valid_file: valid2id.txt wandb: valid_interval: 100 DEBUG:cmd:2024-05-22 15:14:47:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:cmd:2024-05-22 15:14:47:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-05-22 15:14:47:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-05-22 15:14:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:14:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:cmd:2024-05-22 15:14:48:Popen(['git', 'cat-file', '--batch-check'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=, shell=False, universal_newlines=False) wandb: wandb version 0.17.0 is available! To upgrade, please run: wandb: $ pip install wandb --upgrade wandb: Tracking run with wandb version 0.16.6 wandb: Run data is saved locally in /home/luyanfeng/my_code/github/pybind11-OpenKE/examples/RGCN/wandb/run-20240522_151448-7jqvkcn1 wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run peach-sweep-3 wandb: โญ๏ธ View project at https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: ๐Ÿงน View sweep at https://wandb.ai/3555028709/pybind11-ke-sweeps/sweeps/m256vw5a wandb: ๐Ÿš€ View run at https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/7jqvkcn1 DEBUG:connectionpool:2024-05-22 15:14:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:14:56:[cuda:0] Initialization completed, start model training. DEBUG:connectionpool:2024-05-22 15:14:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:15:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:15:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:15:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:15:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:15:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:15:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:15:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:15:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:15:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:15:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:15:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:16:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:16:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:16:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:16:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:16:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:16:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:16:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:16:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:16:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:16:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:16:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:16:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:17:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:17:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:17:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:17:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:17:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:17:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:17:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:17:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:17:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:17:50:[cuda:0] Epoch 100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 15:17:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:17:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:18:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:18:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:18:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:18:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:18:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:18:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:18:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:18:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:18:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:18:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:18:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:19:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:19:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:19:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:19:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:19:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:19:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:19:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:19:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:19:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:19:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:19:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:19:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:20:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:20:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:20:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:20:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:20:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:20:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:20:32:mr: 4299.299 INFO:Trainer:2024-05-22 15:20:32:mrr: 0.037 INFO:Trainer:2024-05-22 15:20:32:hits@1: 0.021 INFO:Trainer:2024-05-22 15:20:32:hits@3: 0.034 INFO:Trainer:2024-05-22 15:20:32:hits@10: 0.064 INFO:Trainer:2024-05-22 15:20:32:mr_type: 348.489 INFO:Trainer:2024-05-22 15:20:32:mrr_type: 0.179 INFO:Trainer:2024-05-22 15:20:32:hits@1_type: 0.123 INFO:Trainer:2024-05-22 15:20:32:hits@3_type: 0.187 INFO:Trainer:2024-05-22 15:20:32:hits@10_type: 0.288 INFO:EarlyStopping:2024-05-22 15:20:32:Validation score improved (-inf --> 0.064000). Saving model ... INFO:Trainer:2024-05-22 15:20:32:[cuda:0] Epoch [ 100/10000] | Batchsize: 70000 | loss: 1.577158 | 1.74569 seconds/epoch DEBUG:connectionpool:2024-05-22 15:20:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:20:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:20:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:20:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:20:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:21:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:21:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:21:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:21:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:21:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:21:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:21:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:21:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:21:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:21:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:21:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:22:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:22:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:22:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:22:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:22:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:22:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:22:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:22:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:22:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:22:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:22:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:22:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:23:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:23:05:[cuda:0] Epoch 200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 15:23:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:23:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:23:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:23:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:23:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:23:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:23:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:23:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:23:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:23:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:24:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:24:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:24:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:24:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:24:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:24:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:24:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:24:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:24:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:24:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:24:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:24:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:25:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:25:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:25:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:25:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:25:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:25:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:25:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:25:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:25:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:25:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:25:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:26:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:26:03:mr: 3676.055 INFO:Trainer:2024-05-22 15:26:03:mrr: 0.056 INFO:Trainer:2024-05-22 15:26:03:hits@1: 0.031 INFO:Trainer:2024-05-22 15:26:03:hits@3: 0.057 INFO:Trainer:2024-05-22 15:26:03:hits@10: 0.111 INFO:Trainer:2024-05-22 15:26:03:mr_type: 335.881 INFO:Trainer:2024-05-22 15:26:03:mrr_type: 0.189 INFO:Trainer:2024-05-22 15:26:03:hits@1_type: 0.128 INFO:Trainer:2024-05-22 15:26:03:hits@3_type: 0.201 INFO:Trainer:2024-05-22 15:26:03:hits@10_type: 0.31 INFO:EarlyStopping:2024-05-22 15:26:03:Validation score improved (0.064000 --> 0.111000). Saving model ... INFO:Trainer:2024-05-22 15:26:03:[cuda:0] Epoch [ 200/10000] | Batchsize: 70000 | loss: 1.520013 | 2.44814 seconds/epoch DEBUG:connectionpool:2024-05-22 15:26:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:26:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:26:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:26:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:26:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:26:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:26:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:26:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:26:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:26:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:27:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:27:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:27:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:27:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:27:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:27:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:27:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:27:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:27:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:27:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:27:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:27:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:28:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:28:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:28:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:28:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:28:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:28:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:28:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:28:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:28:41:[cuda:0] Epoch 300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 15:28:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:28:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:28:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:29:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:29:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:29:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:29:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:29:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:29:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:29:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:29:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:29:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:29:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:29:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:29:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:30:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:30:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:30:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:30:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:30:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:30:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:30:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:30:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:30:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:30:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:30:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:31:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:31:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:31:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:31:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:31:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:31:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:31:29:mr: 3210.485 INFO:Trainer:2024-05-22 15:31:29:mrr: 0.076 INFO:Trainer:2024-05-22 15:31:29:hits@1: 0.041 INFO:Trainer:2024-05-22 15:31:29:hits@3: 0.083 INFO:Trainer:2024-05-22 15:31:29:hits@10: 0.145 INFO:Trainer:2024-05-22 15:31:29:mr_type: 328.833 INFO:Trainer:2024-05-22 15:31:29:mrr_type: 0.193 INFO:Trainer:2024-05-22 15:31:29:hits@1_type: 0.13 INFO:Trainer:2024-05-22 15:31:29:hits@3_type: 0.204 INFO:Trainer:2024-05-22 15:31:29:hits@10_type: 0.316 INFO:EarlyStopping:2024-05-22 15:31:29:Validation score improved (0.111000 --> 0.145000). Saving model ... INFO:Trainer:2024-05-22 15:31:29:[cuda:0] Epoch [ 300/10000] | Batchsize: 70000 | loss: 1.466441 | 2.75005 seconds/epoch DEBUG:connectionpool:2024-05-22 15:31:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:31:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:31:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:31:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:31:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:32:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:32:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:32:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:32:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:32:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:32:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:32:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:32:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:32:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:32:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:32:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:32:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:33:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:33:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:33:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:33:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:33:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:33:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:33:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:33:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:33:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:33:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:33:56:[cuda:0] Epoch 400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 15:33:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:34:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:34:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:34:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:34:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:34:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:34:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:34:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:34:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:34:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:34:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:34:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:35:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:35:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:35:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:35:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:35:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:35:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:35:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:35:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:35:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:35:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:35:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:35:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:36:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:36:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:36:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:36:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:36:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:36:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:36:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:36:38:mr: 2770.803 INFO:Trainer:2024-05-22 15:36:38:mrr: 0.086 INFO:Trainer:2024-05-22 15:36:38:hits@1: 0.048 INFO:Trainer:2024-05-22 15:36:38:hits@3: 0.094 INFO:Trainer:2024-05-22 15:36:38:hits@10: 0.163 INFO:Trainer:2024-05-22 15:36:38:mr_type: 325.73 INFO:Trainer:2024-05-22 15:36:38:mrr_type: 0.19 INFO:Trainer:2024-05-22 15:36:38:hits@1_type: 0.125 INFO:Trainer:2024-05-22 15:36:38:hits@3_type: 0.202 INFO:Trainer:2024-05-22 15:36:38:hits@10_type: 0.316 INFO:EarlyStopping:2024-05-22 15:36:38:Validation score improved (0.145000 --> 0.163000). Saving model ... INFO:Trainer:2024-05-22 15:36:38:[cuda:0] Epoch [ 400/10000] | Batchsize: 70000 | loss: 1.417898 | 2.85078 seconds/epoch DEBUG:connectionpool:2024-05-22 15:36:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:36:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:36:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:36:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:37:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:37:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:37:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:37:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:37:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:37:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:37:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:37:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:37:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:37:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:37:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:37:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:38:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:38:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:38:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:38:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:38:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:38:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:38:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:38:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:38:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:38:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:38:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:39:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:39:05:[cuda:0] Epoch 500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 15:39:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:39:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:39:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:39:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:39:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:39:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:39:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:39:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:39:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:39:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:40:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:40:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:40:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:40:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:40:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:40:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:40:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:40:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:40:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:40:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:40:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:40:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:41:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:41:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:41:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:41:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:41:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:41:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:41:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:41:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:41:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:41:47:mr: 2333.828 INFO:Trainer:2024-05-22 15:41:47:mrr: 0.102 INFO:Trainer:2024-05-22 15:41:47:hits@1: 0.06 INFO:Trainer:2024-05-22 15:41:47:hits@3: 0.109 INFO:Trainer:2024-05-22 15:41:47:hits@10: 0.186 INFO:Trainer:2024-05-22 15:41:47:mr_type: 322.873 INFO:Trainer:2024-05-22 15:41:47:mrr_type: 0.195 INFO:Trainer:2024-05-22 15:41:47:hits@1_type: 0.132 INFO:Trainer:2024-05-22 15:41:47:hits@3_type: 0.207 INFO:Trainer:2024-05-22 15:41:47:hits@10_type: 0.322 INFO:EarlyStopping:2024-05-22 15:41:47:Validation score improved (0.163000 --> 0.186000). Saving model ... INFO:Trainer:2024-05-22 15:41:47:[cuda:0] Epoch [ 500/10000] | Batchsize: 70000 | loss: 1.371727 | 2.89806 seconds/epoch DEBUG:connectionpool:2024-05-22 15:41:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:41:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:42:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:42:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:42:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:42:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:42:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:42:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:42:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:42:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:42:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:42:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:42:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:42:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:43:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:43:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:43:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:43:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:43:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:43:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:43:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:43:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:43:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:43:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:43:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:44:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:44:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:44:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:44:15:[cuda:0] Epoch 600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 15:44:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:44:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:44:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:44:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:44:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:44:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:44:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:44:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:45:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:45:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:45:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:45:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:45:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:45:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:45:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:45:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:45:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:45:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:45:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:45:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:46:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:46:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:46:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:46:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:46:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:46:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:46:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:46:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:46:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:46:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:46:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:46:57:mr: 1954.306 INFO:Trainer:2024-05-22 15:46:57:mrr: 0.113 INFO:Trainer:2024-05-22 15:46:57:hits@1: 0.066 INFO:Trainer:2024-05-22 15:46:57:hits@3: 0.124 INFO:Trainer:2024-05-22 15:46:57:hits@10: 0.205 INFO:Trainer:2024-05-22 15:46:57:mr_type: 319.787 INFO:Trainer:2024-05-22 15:46:57:mrr_type: 0.196 INFO:Trainer:2024-05-22 15:46:57:hits@1_type: 0.129 INFO:Trainer:2024-05-22 15:46:57:hits@3_type: 0.21 INFO:Trainer:2024-05-22 15:46:57:hits@10_type: 0.325 INFO:EarlyStopping:2024-05-22 15:46:57:Validation score improved (0.186000 --> 0.205000). Saving model ... INFO:Trainer:2024-05-22 15:46:57:[cuda:0] Epoch [ 600/10000] | Batchsize: 70000 | loss: 1.325474 | 2.93228 seconds/epoch DEBUG:connectionpool:2024-05-22 15:47:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:47:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:47:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:47:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:47:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:47:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:47:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:47:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:47:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:47:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:47:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:47:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:48:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:48:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:48:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:48:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:48:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:48:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:48:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:48:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:48:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:48:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:48:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:49:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:49:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:49:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:49:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:49:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:49:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:49:31:[cuda:0] Epoch 700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 15:49:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:49:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:49:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:49:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:49:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:50:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:50:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:50:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:50:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:50:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:50:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:50:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:50:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:50:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:50:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:50:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:50:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:51:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:51:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:51:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:51:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:51:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:51:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:51:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:51:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:51:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:51:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:51:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:52:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:52:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:52:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:52:13:mr: 1647.273 INFO:Trainer:2024-05-22 15:52:13:mrr: 0.124 INFO:Trainer:2024-05-22 15:52:13:hits@1: 0.076 INFO:Trainer:2024-05-22 15:52:13:hits@3: 0.133 INFO:Trainer:2024-05-22 15:52:13:hits@10: 0.219 INFO:Trainer:2024-05-22 15:52:13:mr_type: 316.195 INFO:Trainer:2024-05-22 15:52:13:mrr_type: 0.199 INFO:Trainer:2024-05-22 15:52:13:hits@1_type: 0.133 INFO:Trainer:2024-05-22 15:52:13:hits@3_type: 0.212 INFO:Trainer:2024-05-22 15:52:13:hits@10_type: 0.329 INFO:EarlyStopping:2024-05-22 15:52:13:Validation score improved (0.205000 --> 0.219000). Saving model ... INFO:Trainer:2024-05-22 15:52:13:[cuda:0] Epoch [ 700/10000] | Batchsize: 70000 | loss: 1.280761 | 2.96504 seconds/epoch DEBUG:connectionpool:2024-05-22 15:52:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:52:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:52:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:52:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:52:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:52:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:52:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:52:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:52:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:53:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:53:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:53:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:53:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:53:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:53:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:53:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:53:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:53:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:53:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:53:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:54:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:54:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:54:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:54:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:54:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:54:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:54:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:54:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:54:41:[cuda:0] Epoch 800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 15:54:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:54:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:54:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:55:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:55:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:55:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:55:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:55:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:55:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:55:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:55:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:55:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:55:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:55:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:55:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:56:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:56:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:56:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:56:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:56:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:56:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:56:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:56:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:56:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:56:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:56:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:57:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:57:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:57:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:57:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:57:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:57:23:mr: 1415.627 INFO:Trainer:2024-05-22 15:57:23:mrr: 0.13 INFO:Trainer:2024-05-22 15:57:23:hits@1: 0.079 INFO:Trainer:2024-05-22 15:57:23:hits@3: 0.142 INFO:Trainer:2024-05-22 15:57:23:hits@10: 0.229 INFO:Trainer:2024-05-22 15:57:23:mr_type: 312.775 INFO:Trainer:2024-05-22 15:57:23:mrr_type: 0.199 INFO:Trainer:2024-05-22 15:57:23:hits@1_type: 0.132 INFO:Trainer:2024-05-22 15:57:23:hits@3_type: 0.214 INFO:Trainer:2024-05-22 15:57:23:hits@10_type: 0.33 INFO:EarlyStopping:2024-05-22 15:57:23:Validation score improved (0.219000 --> 0.229000). Saving model ... INFO:Trainer:2024-05-22 15:57:23:[cuda:0] Epoch [ 800/10000] | Batchsize: 70000 | loss: 1.240420 | 2.98105 seconds/epoch DEBUG:connectionpool:2024-05-22 15:57:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:57:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:57:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:57:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:57:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:57:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:57:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:58:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:58:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:58:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:58:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:58:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:58:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:58:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:58:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:58:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:58:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:58:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:59:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:59:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:59:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:59:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:59:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:59:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:59:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:59:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:59:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 15:59:48:[cuda:0] Epoch 900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 15:59:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 15:59:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:00:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:00:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:00:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:00:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:00:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:00:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:00:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:00:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:00:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:00:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:00:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:00:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:01:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:01:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:01:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:01:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:01:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:01:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:01:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:01:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:01:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:01:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:01:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:02:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:02:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:02:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:02:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:02:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:02:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:02:30:mr: 1237.209 INFO:Trainer:2024-05-22 16:02:30:mrr: 0.138 INFO:Trainer:2024-05-22 16:02:30:hits@1: 0.084 INFO:Trainer:2024-05-22 16:02:30:hits@3: 0.15 INFO:Trainer:2024-05-22 16:02:30:hits@10: 0.242 INFO:Trainer:2024-05-22 16:02:30:mr_type: 308.869 INFO:Trainer:2024-05-22 16:02:30:mrr_type: 0.202 INFO:Trainer:2024-05-22 16:02:30:hits@1_type: 0.134 INFO:Trainer:2024-05-22 16:02:30:hits@3_type: 0.217 INFO:Trainer:2024-05-22 16:02:30:hits@10_type: 0.336 INFO:EarlyStopping:2024-05-22 16:02:30:Validation score improved (0.229000 --> 0.242000). Saving model ... INFO:Trainer:2024-05-22 16:02:31:[cuda:0] Epoch [ 900/10000] | Batchsize: 70000 | loss: 1.202320 | 2.99175 seconds/epoch DEBUG:connectionpool:2024-05-22 16:02:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:02:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:02:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:02:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:02:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:02:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:03:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:03:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:03:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:03:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:03:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:03:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:03:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:03:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:03:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:03:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:03:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:04:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:04:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:04:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:04:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:04:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:04:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:04:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:04:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:04:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:04:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:04:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:04:58:[cuda:0] Epoch 1000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 16:05:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:05:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:05:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:05:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:05:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:05:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:05:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:05:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:05:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:05:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:05:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:05:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:06:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:06:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:06:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:06:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:06:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:06:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:06:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:06:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:06:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:06:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:06:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:07:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:07:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:07:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:07:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:07:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:07:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:07:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:07:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:07:40:mr: 1102.951 INFO:Trainer:2024-05-22 16:07:40:mrr: 0.142 INFO:Trainer:2024-05-22 16:07:40:hits@1: 0.086 INFO:Trainer:2024-05-22 16:07:40:hits@3: 0.154 INFO:Trainer:2024-05-22 16:07:40:hits@10: 0.25 INFO:Trainer:2024-05-22 16:07:40:mr_type: 305.399 INFO:Trainer:2024-05-22 16:07:40:mrr_type: 0.201 INFO:Trainer:2024-05-22 16:07:40:hits@1_type: 0.133 INFO:Trainer:2024-05-22 16:07:40:hits@3_type: 0.217 INFO:Trainer:2024-05-22 16:07:40:hits@10_type: 0.336 INFO:EarlyStopping:2024-05-22 16:07:40:Validation score improved (0.242000 --> 0.250000). Saving model ... INFO:Trainer:2024-05-22 16:07:40:[cuda:0] Epoch [1000/10000] | Batchsize: 70000 | loss: 1.170018 | 3.00228 seconds/epoch DEBUG:connectionpool:2024-05-22 16:07:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:07:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:07:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:07:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:08:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:08:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:08:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:08:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:08:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:08:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:08:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:08:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:08:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:08:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:08:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:09:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:09:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:09:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:09:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:09:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:09:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:09:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:09:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:09:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:09:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:09:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:10:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:10:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:10:08:[cuda:0] Epoch 1100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 16:10:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:10:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:10:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:10:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:10:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:10:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:10:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:10:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:10:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:10:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:11:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:11:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:11:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:11:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:11:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:11:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:11:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:11:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:11:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:11:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:11:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:12:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:12:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:12:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:12:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:12:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:12:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:12:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:12:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:12:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:12:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:12:49:mr: 994.573 INFO:Trainer:2024-05-22 16:12:49:mrr: 0.145 INFO:Trainer:2024-05-22 16:12:49:hits@1: 0.088 INFO:Trainer:2024-05-22 16:12:49:hits@3: 0.157 INFO:Trainer:2024-05-22 16:12:49:hits@10: 0.256 INFO:Trainer:2024-05-22 16:12:49:mr_type: 302.304 INFO:Trainer:2024-05-22 16:12:49:mrr_type: 0.2 INFO:Trainer:2024-05-22 16:12:49:hits@1_type: 0.132 INFO:Trainer:2024-05-22 16:12:49:hits@3_type: 0.215 INFO:Trainer:2024-05-22 16:12:49:hits@10_type: 0.337 INFO:EarlyStopping:2024-05-22 16:12:49:Validation score improved (0.250000 --> 0.256000). Saving model ... INFO:Trainer:2024-05-22 16:12:50:[cuda:0] Epoch [1100/10000] | Batchsize: 70000 | loss: 1.146386 | 3.01072 seconds/epoch DEBUG:connectionpool:2024-05-22 16:12:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:12:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:13:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:13:08:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-22 16:13:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:13:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:13:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:13:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:13:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:13:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:13:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:13:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:13:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:13:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:14:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:14:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:14:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:14:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:14:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:14:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:14:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:14:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:14:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:14:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:14:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:15:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:15:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:15:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:15:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:15:21:[cuda:0] Epoch 1200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 16:15:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:15:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:15:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:15:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:15:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:15:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:15:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:16:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:16:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:16:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:16:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:16:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:16:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:16:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:16:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:16:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:16:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:16:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:16:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:17:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:17:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:17:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:17:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:17:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:17:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:17:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:17:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:17:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:17:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:17:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:18:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:18:04:mr: 908.769 INFO:Trainer:2024-05-22 16:18:04:mrr: 0.146 INFO:Trainer:2024-05-22 16:18:04:hits@1: 0.088 INFO:Trainer:2024-05-22 16:18:04:hits@3: 0.16 INFO:Trainer:2024-05-22 16:18:04:hits@10: 0.26 INFO:Trainer:2024-05-22 16:18:04:mr_type: 298.984 INFO:Trainer:2024-05-22 16:18:04:mrr_type: 0.198 INFO:Trainer:2024-05-22 16:18:04:hits@1_type: 0.129 INFO:Trainer:2024-05-22 16:18:04:hits@3_type: 0.214 INFO:Trainer:2024-05-22 16:18:04:hits@10_type: 0.335 INFO:EarlyStopping:2024-05-22 16:18:04:Validation score improved (0.256000 --> 0.260000). Saving model ... INFO:Trainer:2024-05-22 16:18:04:[cuda:0] Epoch [1200/10000] | Batchsize: 70000 | loss: 1.122949 | 3.02138 seconds/epoch DEBUG:connectionpool:2024-05-22 16:18:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:18:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:18:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:18:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:18:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:18:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:18:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:18:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:18:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:18:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:18:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:19:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:19:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:19:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:19:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:19:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:19:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:19:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:19:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:19:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:19:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:19:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:20:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:20:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:20:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:20:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:20:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:20:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:20:30:[cuda:0] Epoch 1300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 16:20:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:20:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:20:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:20:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:20:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:21:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:21:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:21:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:21:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:21:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:21:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:21:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:21:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:21:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:21:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:21:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:21:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:22:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:22:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:22:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:22:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:22:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:22:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:22:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:22:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:22:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:22:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:22:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:23:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:23:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:23:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:23:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:23:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:23:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:23:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:23:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:23:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:23:48:mr: 839.092 INFO:Trainer:2024-05-22 16:23:48:mrr: 0.152 INFO:Trainer:2024-05-22 16:23:48:hits@1: 0.093 INFO:Trainer:2024-05-22 16:23:48:hits@3: 0.166 INFO:Trainer:2024-05-22 16:23:48:hits@10: 0.266 INFO:Trainer:2024-05-22 16:23:48:mr_type: 296.286 INFO:Trainer:2024-05-22 16:23:48:mrr_type: 0.201 INFO:Trainer:2024-05-22 16:23:48:hits@1_type: 0.133 INFO:Trainer:2024-05-22 16:23:48:hits@3_type: 0.216 INFO:Trainer:2024-05-22 16:23:48:hits@10_type: 0.336 INFO:EarlyStopping:2024-05-22 16:23:48:Validation score improved (0.260000 --> 0.266000). Saving model ... INFO:Trainer:2024-05-22 16:23:48:[cuda:0] Epoch [1300/10000] | Batchsize: 70000 | loss: 1.103337 | 3.02667 seconds/epoch DEBUG:connectionpool:2024-05-22 16:23:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:23:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:24:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:24:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:24:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:24:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:24:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:24:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:24:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:24:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:24:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:24:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:24:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:24:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:25:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:25:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:25:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:25:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:25:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:25:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:25:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:25:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:25:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:25:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:25:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:26:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:26:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:26:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:26:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:26:22:[cuda:0] Epoch 1400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 16:26:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:26:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:26:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:26:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:26:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:26:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:26:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:27:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:27:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:27:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:27:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:27:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:27:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:27:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:27:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:27:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:27:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:27:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:27:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:28:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:28:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:28:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:28:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:28:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:28:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:28:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:28:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:28:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:28:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:28:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:29:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:29:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:29:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:29:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:29:21:mr: 781.996 INFO:Trainer:2024-05-22 16:29:21:mrr: 0.156 INFO:Trainer:2024-05-22 16:29:21:hits@1: 0.096 INFO:Trainer:2024-05-22 16:29:21:hits@3: 0.171 INFO:Trainer:2024-05-22 16:29:21:hits@10: 0.274 INFO:Trainer:2024-05-22 16:29:21:mr_type: 293.308 INFO:Trainer:2024-05-22 16:29:21:mrr_type: 0.203 INFO:Trainer:2024-05-22 16:29:21:hits@1_type: 0.134 INFO:Trainer:2024-05-22 16:29:21:hits@3_type: 0.219 INFO:Trainer:2024-05-22 16:29:21:hits@10_type: 0.34 INFO:EarlyStopping:2024-05-22 16:29:21:Validation score improved (0.266000 --> 0.274000). Saving model ... INFO:Trainer:2024-05-22 16:29:21:[cuda:0] Epoch [1400/10000] | Batchsize: 70000 | loss: 1.085414 | 3.06170 seconds/epoch DEBUG:connectionpool:2024-05-22 16:29:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:29:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:29:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:29:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:29:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:29:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:29:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:29:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:30:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:30:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:30:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:30:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:30:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:30:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:30:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:30:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:30:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:30:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:30:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:31:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:31:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:31:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:31:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:31:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:31:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:31:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:31:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:31:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:31:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:31:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:32:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:32:05:[cuda:0] Epoch 1500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 16:32:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:32:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:32:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:32:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:32:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:32:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:32:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:32:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:32:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:32:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:32:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:33:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:33:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:33:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:33:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:33:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:33:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:33:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:33:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:33:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:33:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:33:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:34:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:34:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:34:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:34:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:34:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:34:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:34:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:34:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:34:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:34:47:mr: 734.606 INFO:Trainer:2024-05-22 16:34:47:mrr: 0.156 INFO:Trainer:2024-05-22 16:34:47:hits@1: 0.095 INFO:Trainer:2024-05-22 16:34:47:hits@3: 0.171 INFO:Trainer:2024-05-22 16:34:47:hits@10: 0.277 INFO:Trainer:2024-05-22 16:34:47:mr_type: 290.499 INFO:Trainer:2024-05-22 16:34:47:mrr_type: 0.2 INFO:Trainer:2024-05-22 16:34:47:hits@1_type: 0.131 INFO:Trainer:2024-05-22 16:34:47:hits@3_type: 0.216 INFO:Trainer:2024-05-22 16:34:47:hits@10_type: 0.338 INFO:EarlyStopping:2024-05-22 16:34:47:Validation score improved (0.274000 --> 0.277000). Saving model ... INFO:Trainer:2024-05-22 16:34:47:[cuda:0] Epoch [1500/10000] | Batchsize: 70000 | loss: 1.068741 | 3.08594 seconds/epoch DEBUG:connectionpool:2024-05-22 16:34:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:34:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:34:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:35:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:35:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:35:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:35:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:35:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:35:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:35:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:35:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:35:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:35:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:35:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:36:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:36:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:36:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:36:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:36:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:36:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:36:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:36:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:36:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:36:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:36:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:37:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:37:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:37:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:37:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:37:19:[cuda:0] Epoch 1600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 16:37:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:37:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:37:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:37:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:37:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:37:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:37:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:38:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:38:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:38:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:38:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:38:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:38:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:38:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:38:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:38:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:38:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:38:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:38:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:39:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:39:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:39:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:39:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:39:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:39:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:39:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:39:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:39:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:39:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:39:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:40:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:40:02:mr: 695.432 INFO:Trainer:2024-05-22 16:40:02:mrr: 0.161 INFO:Trainer:2024-05-22 16:40:02:hits@1: 0.098 INFO:Trainer:2024-05-22 16:40:02:hits@3: 0.177 INFO:Trainer:2024-05-22 16:40:02:hits@10: 0.282 INFO:Trainer:2024-05-22 16:40:02:mr_type: 288.199 INFO:Trainer:2024-05-22 16:40:02:mrr_type: 0.203 INFO:Trainer:2024-05-22 16:40:02:hits@1_type: 0.133 INFO:Trainer:2024-05-22 16:40:02:hits@3_type: 0.218 INFO:Trainer:2024-05-22 16:40:02:hits@10_type: 0.34 INFO:EarlyStopping:2024-05-22 16:40:02:Validation score improved (0.277000 --> 0.282000). Saving model ... INFO:Trainer:2024-05-22 16:40:02:[cuda:0] Epoch [1600/10000] | Batchsize: 70000 | loss: 1.052572 | 3.08973 seconds/epoch DEBUG:connectionpool:2024-05-22 16:40:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:40:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:40:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:40:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:40:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:40:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:40:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:40:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:40:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:40:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:40:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:41:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:41:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:41:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:41:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:41:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:41:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:41:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:41:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:41:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:41:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:41:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:42:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:42:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:42:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:42:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:42:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:42:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:42:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:42:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:42:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:42:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:42:51:[cuda:0] Epoch 1700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 16:42:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:43:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:43:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:43:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:43:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:43:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:43:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:43:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:43:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:43:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:43:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:43:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:43:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:44:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:44:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:44:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:44:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:44:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:44:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:44:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:44:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:44:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:44:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:44:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:45:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:45:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:45:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:45:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:45:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:45:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:45:34:mr: 662.316 INFO:Trainer:2024-05-22 16:45:34:mrr: 0.16 INFO:Trainer:2024-05-22 16:45:34:hits@1: 0.096 INFO:Trainer:2024-05-22 16:45:34:hits@3: 0.177 INFO:Trainer:2024-05-22 16:45:34:hits@10: 0.285 INFO:Trainer:2024-05-22 16:45:34:mr_type: 285.699 INFO:Trainer:2024-05-22 16:45:34:mrr_type: 0.2 INFO:Trainer:2024-05-22 16:45:34:hits@1_type: 0.129 INFO:Trainer:2024-05-22 16:45:34:hits@3_type: 0.216 INFO:Trainer:2024-05-22 16:45:34:hits@10_type: 0.341 INFO:EarlyStopping:2024-05-22 16:45:34:Validation score improved (0.282000 --> 0.285000). Saving model ... INFO:Trainer:2024-05-22 16:45:34:[cuda:0] Epoch [1700/10000] | Batchsize: 70000 | loss: 1.038713 | 3.10335 seconds/epoch DEBUG:connectionpool:2024-05-22 16:45:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:45:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:45:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:45:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:45:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:46:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:46:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:46:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:46:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:46:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:46:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:46:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:46:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:46:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:46:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:46:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:46:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:47:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:47:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:47:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:47:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:47:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:47:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:47:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:47:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:47:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:47:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:47:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:48:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:48:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:48:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:48:17:[cuda:0] Epoch 1800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 16:48:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:48:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:48:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:48:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:48:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:48:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:48:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:48:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:49:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:49:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:49:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:49:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:49:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:49:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:49:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:49:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:49:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:49:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:49:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:49:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:50:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:50:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:50:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:50:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:50:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:50:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:50:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:50:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:50:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:50:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:50:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:51:00:mr: 636.397 INFO:Trainer:2024-05-22 16:51:00:mrr: 0.162 INFO:Trainer:2024-05-22 16:51:00:hits@1: 0.097 INFO:Trainer:2024-05-22 16:51:00:hits@3: 0.179 INFO:Trainer:2024-05-22 16:51:00:hits@10: 0.286 INFO:Trainer:2024-05-22 16:51:00:mr_type: 283.748 INFO:Trainer:2024-05-22 16:51:00:mrr_type: 0.2 INFO:Trainer:2024-05-22 16:51:00:hits@1_type: 0.129 INFO:Trainer:2024-05-22 16:51:00:hits@3_type: 0.216 INFO:Trainer:2024-05-22 16:51:00:hits@10_type: 0.34 INFO:EarlyStopping:2024-05-22 16:51:00:Validation score improved (0.285000 --> 0.286000). Saving model ... INFO:Trainer:2024-05-22 16:51:00:[cuda:0] Epoch [1800/10000] | Batchsize: 70000 | loss: 1.028013 | 3.11186 seconds/epoch DEBUG:connectionpool:2024-05-22 16:51:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:51:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:51:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:51:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:51:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:51:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:51:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:51:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:51:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:51:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:51:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:51:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:52:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:52:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:52:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:52:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:52:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:52:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:52:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:52:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:52:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:52:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:52:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:53:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:53:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:53:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:53:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:53:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:53:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:53:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:53:39:[cuda:0] Epoch 1900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 16:53:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:53:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:53:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:53:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:54:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:54:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:54:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:54:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:54:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:54:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:54:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:54:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:54:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:54:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:54:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:54:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:55:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:55:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:55:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:55:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:55:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:55:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:55:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:55:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:55:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:55:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:55:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:56:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:56:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:56:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:56:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:56:22:mr: 612.795 INFO:Trainer:2024-05-22 16:56:22:mrr: 0.164 INFO:Trainer:2024-05-22 16:56:22:hits@1: 0.099 INFO:Trainer:2024-05-22 16:56:22:hits@3: 0.182 INFO:Trainer:2024-05-22 16:56:22:hits@10: 0.291 INFO:Trainer:2024-05-22 16:56:22:mr_type: 281.73 INFO:Trainer:2024-05-22 16:56:22:mrr_type: 0.201 INFO:Trainer:2024-05-22 16:56:22:hits@1_type: 0.13 INFO:Trainer:2024-05-22 16:56:22:hits@3_type: 0.218 INFO:Trainer:2024-05-22 16:56:22:hits@10_type: 0.342 INFO:EarlyStopping:2024-05-22 16:56:22:Validation score improved (0.286000 --> 0.291000). Saving model ... INFO:Trainer:2024-05-22 16:56:22:[cuda:0] Epoch [1900/10000] | Batchsize: 70000 | loss: 1.015827 | 3.11774 seconds/epoch DEBUG:connectionpool:2024-05-22 16:56:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:56:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:56:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:56:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:56:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:56:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:56:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:57:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:57:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:57:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:57:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:57:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:57:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:57:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:57:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:57:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:57:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:57:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:57:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:58:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:58:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:58:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:58:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:58:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:58:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:58:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:58:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:58:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:58:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:58:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:59:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:59:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:59:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 16:59:14:[cuda:0] Epoch 2000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 16:59:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:59:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:59:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:59:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:59:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:59:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:59:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 16:59:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:00:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:00:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:00:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:00:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:00:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:00:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:00:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:00:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:00:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:00:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:00:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:00:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:01:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:01:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:01:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:01:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:01:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:01:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:01:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:01:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:01:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:01:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:01:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:01:56:mr: 594.133 INFO:Trainer:2024-05-22 17:01:56:mrr: 0.166 INFO:Trainer:2024-05-22 17:01:56:hits@1: 0.101 INFO:Trainer:2024-05-22 17:01:56:hits@3: 0.183 INFO:Trainer:2024-05-22 17:01:56:hits@10: 0.293 INFO:Trainer:2024-05-22 17:01:56:mr_type: 280.047 INFO:Trainer:2024-05-22 17:01:56:mrr_type: 0.202 INFO:Trainer:2024-05-22 17:01:56:hits@1_type: 0.131 INFO:Trainer:2024-05-22 17:01:56:hits@3_type: 0.218 INFO:Trainer:2024-05-22 17:01:56:hits@10_type: 0.342 INFO:EarlyStopping:2024-05-22 17:01:56:Validation score improved (0.291000 --> 0.293000). Saving model ... INFO:Trainer:2024-05-22 17:01:57:[cuda:0] Epoch [2000/10000] | Batchsize: 70000 | loss: 1.005454 | 3.12903 seconds/epoch DEBUG:connectionpool:2024-05-22 17:02:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:02:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:02:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:02:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:02:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:02:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:02:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:02:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:02:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:02:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:02:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:03:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:03:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:03:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:03:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:03:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:03:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:03:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:03:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:03:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:03:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:03:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:03:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:04:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:04:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:04:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:04:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:04:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:04:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:04:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:04:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:04:41:[cuda:0] Epoch 2100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 17:04:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:04:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:04:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:05:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:05:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:05:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:05:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:05:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:05:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:05:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:05:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:05:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:05:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:05:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:05:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:06:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:06:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:06:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:06:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:06:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:06:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:06:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:06:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:06:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:06:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:06:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:07:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:07:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:07:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:07:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:07:23:mr: 578.33 INFO:Trainer:2024-05-22 17:07:23:mrr: 0.167 INFO:Trainer:2024-05-22 17:07:23:hits@1: 0.102 INFO:Trainer:2024-05-22 17:07:23:hits@3: 0.184 INFO:Trainer:2024-05-22 17:07:23:hits@10: 0.296 INFO:Trainer:2024-05-22 17:07:23:mr_type: 278.406 INFO:Trainer:2024-05-22 17:07:23:mrr_type: 0.203 INFO:Trainer:2024-05-22 17:07:23:hits@1_type: 0.131 INFO:Trainer:2024-05-22 17:07:23:hits@3_type: 0.218 INFO:Trainer:2024-05-22 17:07:23:hits@10_type: 0.344 INFO:EarlyStopping:2024-05-22 17:07:23:Validation score improved (0.293000 --> 0.296000). Saving model ... INFO:Trainer:2024-05-22 17:07:23:[cuda:0] Epoch [2100/10000] | Batchsize: 70000 | loss: 0.997308 | 3.13578 seconds/epoch DEBUG:connectionpool:2024-05-22 17:07:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:07:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:07:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:07:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:07:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:07:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:07:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:08:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:08:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:08:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:08:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:08:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:08:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:08:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:08:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:08:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:08:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:08:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:08:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:09:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:09:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:09:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:09:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:09:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:09:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:09:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:09:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:09:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:09:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:09:54:[cuda:0] Epoch 2200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 17:09:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:10:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:10:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:10:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:10:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:10:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:10:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:10:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:10:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:10:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:10:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:10:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:10:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:11:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:11:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:11:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:11:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:11:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:11:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:11:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:11:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:11:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:11:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:11:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:12:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:12:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:12:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:12:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:12:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:12:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:12:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:12:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:12:44:mr: 564.399 INFO:Trainer:2024-05-22 17:12:44:mrr: 0.17 INFO:Trainer:2024-05-22 17:12:44:hits@1: 0.105 INFO:Trainer:2024-05-22 17:12:44:hits@3: 0.188 INFO:Trainer:2024-05-22 17:12:44:hits@10: 0.299 INFO:Trainer:2024-05-22 17:12:44:mr_type: 276.711 INFO:Trainer:2024-05-22 17:12:44:mrr_type: 0.205 INFO:Trainer:2024-05-22 17:12:44:hits@1_type: 0.133 INFO:Trainer:2024-05-22 17:12:44:hits@3_type: 0.221 INFO:Trainer:2024-05-22 17:12:44:hits@10_type: 0.345 INFO:EarlyStopping:2024-05-22 17:12:44:Validation score improved (0.296000 --> 0.299000). Saving model ... INFO:Trainer:2024-05-22 17:12:44:[cuda:0] Epoch [2200/10000] | Batchsize: 70000 | loss: 0.988011 | 3.13575 seconds/epoch DEBUG:connectionpool:2024-05-22 17:12:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:12:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:12:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:13:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:13:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:13:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:13:16:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-22 17:13:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:13:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:13:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:13:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:13:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:13:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:13:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:13:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:14:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:14:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:14:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:14:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:14:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:14:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:14:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:14:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:14:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:14:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:14:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:14:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:15:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:15:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:15:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:15:17:[cuda:0] Epoch 2300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 17:15:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:15:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:15:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:15:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:15:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:15:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:15:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:15:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:16:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:16:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:16:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:16:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:16:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:16:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:16:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:16:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:16:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:16:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:16:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:16:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:17:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:17:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:17:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:17:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:17:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:17:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:17:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:17:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:17:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:17:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:17:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:17:59:mr: 551.717 INFO:Trainer:2024-05-22 17:17:59:mrr: 0.167 INFO:Trainer:2024-05-22 17:17:59:hits@1: 0.101 INFO:Trainer:2024-05-22 17:17:59:hits@3: 0.185 INFO:Trainer:2024-05-22 17:17:59:hits@10: 0.299 INFO:Trainer:2024-05-22 17:17:59:mr_type: 275.145 INFO:Trainer:2024-05-22 17:17:59:mrr_type: 0.201 INFO:Trainer:2024-05-22 17:17:59:hits@1_type: 0.128 INFO:Trainer:2024-05-22 17:17:59:hits@3_type: 0.218 INFO:Trainer:2024-05-22 17:17:59:hits@10_type: 0.344 INFO:EarlyStopping:2024-05-22 17:17:59:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-05-22 17:17:59:[cuda:0] Epoch [2300/10000] | Batchsize: 70000 | loss: 0.979469 | 3.13985 seconds/epoch DEBUG:connectionpool:2024-05-22 17:18:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:18:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:18:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:18:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:18:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:18:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:18:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:18:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:18:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:18:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:18:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:19:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:19:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:19:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:19:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:19:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:19:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:19:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:19:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:19:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:19:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:19:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:19:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:20:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:20:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:20:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:20:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:20:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:20:28:[cuda:0] Epoch 2400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 17:20:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:20:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:20:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:20:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:20:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:20:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:21:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:21:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:21:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:21:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:21:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:21:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:21:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:21:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:21:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:21:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:21:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:22:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:22:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:22:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:22:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:22:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:22:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:22:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:22:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:22:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:22:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:22:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:22:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:23:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:23:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:23:10:mr: 540.922 INFO:Trainer:2024-05-22 17:23:10:mrr: 0.17 INFO:Trainer:2024-05-22 17:23:10:hits@1: 0.104 INFO:Trainer:2024-05-22 17:23:10:hits@3: 0.187 INFO:Trainer:2024-05-22 17:23:10:hits@10: 0.302 INFO:Trainer:2024-05-22 17:23:10:mr_type: 273.912 INFO:Trainer:2024-05-22 17:23:10:mrr_type: 0.203 INFO:Trainer:2024-05-22 17:23:10:hits@1_type: 0.132 INFO:Trainer:2024-05-22 17:23:10:hits@3_type: 0.219 INFO:Trainer:2024-05-22 17:23:10:hits@10_type: 0.346 INFO:EarlyStopping:2024-05-22 17:23:10:Validation score improved (0.299000 --> 0.302000). Saving model ... INFO:Trainer:2024-05-22 17:23:10:[cuda:0] Epoch [2400/10000] | Batchsize: 70000 | loss: 0.972083 | 3.13850 seconds/epoch DEBUG:connectionpool:2024-05-22 17:23:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:23:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:23:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:23:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:23:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:23:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:23:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:23:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:23:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:24:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:24:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:24:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:24:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:24:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:24:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:24:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:24:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:24:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:24:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:24:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:24:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:25:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:25:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:25:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:25:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:25:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:25:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:25:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:25:39:[cuda:0] Epoch 2500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 17:25:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:25:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:25:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:25:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:26:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:26:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:26:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:26:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:26:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:26:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:26:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:26:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:26:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:26:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:26:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:27:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:27:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:27:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:27:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:27:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:27:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:27:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:27:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:27:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:27:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:27:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:27:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:28:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:28:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:28:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:28:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:28:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:28:29:mr: 531.407 INFO:Trainer:2024-05-22 17:28:29:mrr: 0.171 INFO:Trainer:2024-05-22 17:28:29:hits@1: 0.104 INFO:Trainer:2024-05-22 17:28:29:hits@3: 0.19 INFO:Trainer:2024-05-22 17:28:29:hits@10: 0.303 INFO:Trainer:2024-05-22 17:28:29:mr_type: 272.376 INFO:Trainer:2024-05-22 17:28:29:mrr_type: 0.204 INFO:Trainer:2024-05-22 17:28:29:hits@1_type: 0.131 INFO:Trainer:2024-05-22 17:28:29:hits@3_type: 0.221 INFO:Trainer:2024-05-22 17:28:29:hits@10_type: 0.346 INFO:EarlyStopping:2024-05-22 17:28:29:Validation score improved (0.302000 --> 0.303000). Saving model ... INFO:Trainer:2024-05-22 17:28:29:[cuda:0] Epoch [2500/10000] | Batchsize: 70000 | loss: 0.964495 | 3.13739 seconds/epoch DEBUG:connectionpool:2024-05-22 17:28:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:28:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:28:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:28:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:28:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:28:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:29:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:29:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:29:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:29:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:29:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:29:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:29:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:29:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:29:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:29:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:29:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:30:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:30:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:30:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:30:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:30:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:30:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:30:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:30:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:30:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:30:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:30:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:30:57:[cuda:0] Epoch 2600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 17:30:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:31:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:31:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:31:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:31:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:31:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:31:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:31:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:31:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:31:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:31:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:31:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:32:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:32:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:32:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:32:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:32:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:32:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:32:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:32:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:32:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:32:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:32:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:32:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:33:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:33:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:33:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:33:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:33:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:33:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:33:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:33:39:mr: 522.608 INFO:Trainer:2024-05-22 17:33:39:mrr: 0.175 INFO:Trainer:2024-05-22 17:33:39:hits@1: 0.108 INFO:Trainer:2024-05-22 17:33:39:hits@3: 0.192 INFO:Trainer:2024-05-22 17:33:39:hits@10: 0.305 INFO:Trainer:2024-05-22 17:33:39:mr_type: 271.12 INFO:Trainer:2024-05-22 17:33:39:mrr_type: 0.207 INFO:Trainer:2024-05-22 17:33:39:hits@1_type: 0.135 INFO:Trainer:2024-05-22 17:33:39:hits@3_type: 0.223 INFO:Trainer:2024-05-22 17:33:39:hits@10_type: 0.348 INFO:EarlyStopping:2024-05-22 17:33:39:Validation score improved (0.303000 --> 0.305000). Saving model ... INFO:Trainer:2024-05-22 17:33:39:[cuda:0] Epoch [2600/10000] | Batchsize: 70000 | loss: 0.956928 | 3.13894 seconds/epoch DEBUG:connectionpool:2024-05-22 17:33:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:33:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:33:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:33:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:34:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:34:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:34:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:34:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:34:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:34:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:34:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:34:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:34:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:34:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:34:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:35:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:35:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:35:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:35:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:35:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:35:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:35:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:35:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:35:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:35:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:35:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:35:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:36:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:36:07:[cuda:0] Epoch 2700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 17:36:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:36:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:36:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:36:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:36:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:36:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:36:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:36:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:36:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:36:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:37:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:37:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:37:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:37:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:37:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:37:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:37:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:37:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:37:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:37:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:37:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:38:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:38:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:38:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:38:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:38:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:38:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:38:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:38:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:38:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:38:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:38:49:mr: 514.591 INFO:Trainer:2024-05-22 17:38:49:mrr: 0.173 INFO:Trainer:2024-05-22 17:38:49:hits@1: 0.107 INFO:Trainer:2024-05-22 17:38:49:hits@3: 0.191 INFO:Trainer:2024-05-22 17:38:49:hits@10: 0.304 INFO:Trainer:2024-05-22 17:38:49:mr_type: 269.994 INFO:Trainer:2024-05-22 17:38:49:mrr_type: 0.205 INFO:Trainer:2024-05-22 17:38:49:hits@1_type: 0.133 INFO:Trainer:2024-05-22 17:38:49:hits@3_type: 0.221 INFO:Trainer:2024-05-22 17:38:49:hits@10_type: 0.346 INFO:EarlyStopping:2024-05-22 17:38:49:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-05-22 17:38:49:[cuda:0] Epoch [2700/10000] | Batchsize: 70000 | loss: 0.951770 | 3.13742 seconds/epoch DEBUG:connectionpool:2024-05-22 17:38:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:38:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:39:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:39:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:39:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:39:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:39:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:39:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:39:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:39:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:39:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:39:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:39:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:40:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:40:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:40:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:40:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:40:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:40:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:40:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:40:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:40:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:40:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:40:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:41:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:41:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:41:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:41:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:41:16:[cuda:0] Epoch 2800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 17:41:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:41:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:41:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:41:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:41:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:41:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:41:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:41:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:42:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:42:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:42:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:42:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:42:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:42:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:42:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:42:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:42:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:42:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:42:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:43:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:43:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:43:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:43:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:43:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:43:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:43:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:43:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:43:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:43:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:43:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:43:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:43:58:mr: 507.635 INFO:Trainer:2024-05-22 17:43:58:mrr: 0.175 INFO:Trainer:2024-05-22 17:43:58:hits@1: 0.108 INFO:Trainer:2024-05-22 17:43:58:hits@3: 0.193 INFO:Trainer:2024-05-22 17:43:58:hits@10: 0.307 INFO:Trainer:2024-05-22 17:43:58:mr_type: 268.738 INFO:Trainer:2024-05-22 17:43:58:mrr_type: 0.206 INFO:Trainer:2024-05-22 17:43:58:hits@1_type: 0.134 INFO:Trainer:2024-05-22 17:43:58:hits@3_type: 0.222 INFO:Trainer:2024-05-22 17:43:58:hits@10_type: 0.348 INFO:EarlyStopping:2024-05-22 17:43:58:Validation score improved (0.305000 --> 0.307000). Saving model ... INFO:Trainer:2024-05-22 17:43:58:[cuda:0] Epoch [2800/10000] | Batchsize: 70000 | loss: 0.944106 | 3.13588 seconds/epoch DEBUG:connectionpool:2024-05-22 17:44:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:44:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:44:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:44:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:44:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:44:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:44:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:44:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:44:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:44:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:44:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:45:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:45:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:45:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:45:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:45:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:45:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:45:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:45:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:45:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:45:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:45:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:46:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:46:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:46:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:46:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:46:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:46:26:[cuda:0] Epoch 2900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 17:46:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:46:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:46:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:46:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:46:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:46:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:46:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:47:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:47:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:47:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:47:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:47:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:47:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:47:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:47:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:47:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:47:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:47:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:48:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:48:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:48:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:48:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:48:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:48:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:48:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:48:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:48:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:48:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:48:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:48:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:49:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:49:08:mr: 501.154 INFO:Trainer:2024-05-22 17:49:08:mrr: 0.177 INFO:Trainer:2024-05-22 17:49:08:hits@1: 0.109 INFO:Trainer:2024-05-22 17:49:08:hits@3: 0.195 INFO:Trainer:2024-05-22 17:49:08:hits@10: 0.309 INFO:Trainer:2024-05-22 17:49:08:mr_type: 267.641 INFO:Trainer:2024-05-22 17:49:08:mrr_type: 0.207 INFO:Trainer:2024-05-22 17:49:08:hits@1_type: 0.135 INFO:Trainer:2024-05-22 17:49:08:hits@3_type: 0.224 INFO:Trainer:2024-05-22 17:49:08:hits@10_type: 0.35 INFO:EarlyStopping:2024-05-22 17:49:08:Validation score improved (0.307000 --> 0.309000). Saving model ... INFO:Trainer:2024-05-22 17:49:08:[cuda:0] Epoch [2900/10000] | Batchsize: 70000 | loss: 0.937423 | 3.13460 seconds/epoch DEBUG:connectionpool:2024-05-22 17:49:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:49:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:49:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:49:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:49:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:49:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:49:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:49:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:49:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:49:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:50:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:50:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:50:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:50:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:50:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:50:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:50:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:50:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:50:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:50:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:50:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:51:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:51:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:51:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:51:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:51:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:51:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:51:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:51:36:[cuda:0] Epoch 3000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 17:51:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:51:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:51:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:51:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:52:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:52:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:52:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:52:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:52:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:52:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:52:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:52:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:52:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:52:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:52:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:52:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:53:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:53:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:53:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:53:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:53:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:53:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:53:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:53:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:53:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:53:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:53:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:54:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:54:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:54:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:54:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:54:18:mr: 495.15 INFO:Trainer:2024-05-22 17:54:18:mrr: 0.178 INFO:Trainer:2024-05-22 17:54:18:hits@1: 0.111 INFO:Trainer:2024-05-22 17:54:18:hits@3: 0.195 INFO:Trainer:2024-05-22 17:54:18:hits@10: 0.309 INFO:Trainer:2024-05-22 17:54:18:mr_type: 266.637 INFO:Trainer:2024-05-22 17:54:18:mrr_type: 0.208 INFO:Trainer:2024-05-22 17:54:18:hits@1_type: 0.137 INFO:Trainer:2024-05-22 17:54:18:hits@3_type: 0.224 INFO:Trainer:2024-05-22 17:54:18:hits@10_type: 0.349 INFO:EarlyStopping:2024-05-22 17:54:18:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-05-22 17:54:18:[cuda:0] Epoch [3000/10000] | Batchsize: 70000 | loss: 0.932598 | 3.13336 seconds/epoch DEBUG:connectionpool:2024-05-22 17:54:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:54:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:54:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:54:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:54:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:54:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:54:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:54:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:55:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:55:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:55:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:55:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:55:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:55:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:55:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:55:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:55:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:55:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:55:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:56:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:56:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:56:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:56:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:56:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:56:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:56:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:56:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:56:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:56:46:[cuda:0] Epoch 3100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-22 17:56:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:56:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:57:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:57:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:57:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:57:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:57:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:57:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:57:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:57:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:57:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:57:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:57:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:57:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:58:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:58:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:58:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:58:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:58:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:58:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:58:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:58:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:58:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:58:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:58:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:59:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:59:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:59:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:59:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:59:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:59:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 17:59:29:mr: 489.847 INFO:Trainer:2024-05-22 17:59:29:mrr: 0.176 INFO:Trainer:2024-05-22 17:59:29:hits@1: 0.109 INFO:Trainer:2024-05-22 17:59:29:hits@3: 0.193 INFO:Trainer:2024-05-22 17:59:29:hits@10: 0.308 INFO:Trainer:2024-05-22 17:59:29:mr_type: 265.74 INFO:Trainer:2024-05-22 17:59:29:mrr_type: 0.206 INFO:Trainer:2024-05-22 17:59:29:hits@1_type: 0.134 INFO:Trainer:2024-05-22 17:59:29:hits@3_type: 0.222 INFO:Trainer:2024-05-22 17:59:29:hits@10_type: 0.349 INFO:EarlyStopping:2024-05-22 17:59:29:EarlyStopping counter: 2 / 2 INFO:Trainer:2024-05-22 17:59:29:[cuda:0] Send an early stopping signal INFO:Trainer:2024-05-22 17:59:29:[cuda:0] The model training is completed, taking a total of 9710.46408 seconds. INFO:Trainer:2024-05-22 17:59:29:[cuda:0] Model saved at ./model.pth. INFO:Trainer:2024-05-22 17:59:29:[cuda:0] The model starts evaluating in the test set. DEBUG:connectionpool:2024-05-22 17:59:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:59:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:59:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:59:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:59:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 17:59:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:00:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:00:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:00:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:00:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:00:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:00:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:00:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:00:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:00:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:00:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:00:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:01:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:01:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:01:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:01:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:01:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:01:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:01:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:01:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:01:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:01:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:01:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:01:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:02:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:02:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:02:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:02:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:02:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:02:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-22 18:02:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-22 18:02:38:mr: 501.846 INFO:Trainer:2024-05-22 18:02:38:mrr: 0.172 INFO:Trainer:2024-05-22 18:02:38:hits@1: 0.104 INFO:Trainer:2024-05-22 18:02:38:hits@3: 0.189 INFO:Trainer:2024-05-22 18:02:38:hits@10: 0.304 INFO:Trainer:2024-05-22 18:02:38:mr_type: 272.098 INFO:Trainer:2024-05-22 18:02:38:mrr_type: 0.2 INFO:Trainer:2024-05-22 18:02:38:hits@1_type: 0.128 INFO:Trainer:2024-05-22 18:02:38:hits@3_type: 0.217 INFO:Trainer:2024-05-22 18:02:38:hits@10_type: 0.343 wandb: - 0.014 MB of 0.014 MB uploaded wandb: \ 0.014 MB of 0.014 MB uploaded DEBUG:connectionpool:2024-05-22 18:02:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: | 0.014 MB of 0.014 MB uploaded wandb: / 0.014 MB of 0.241 MB uploaded wandb: - 0.016 MB of 0.241 MB uploaded wandb: \ 0.225 MB of 0.241 MB uploaded wandb: | 0.225 MB of 0.241 MB uploaded DEBUG:connectionpool:2024-05-22 18:02:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: / 0.241 MB of 0.241 MB uploaded wandb: - 0.241 MB of 0.241 MB uploaded wandb: \ 0.241 MB of 0.241 MB uploaded wandb: wandb: wandb: Run history: wandb: duration โ– wandb: test/hits@1 โ– wandb: test/hits@10 โ– wandb: test/hits@10_type โ– wandb: test/hits@1_type โ– wandb: test/hits@3 โ– wandb: test/hits@3_type โ– wandb: test/mr โ– wandb: test/mr_type โ– wandb: test/mrr โ– wandb: test/mrr_type โ– wandb: train/epoch โ–โ–โ–โ–‚โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆ wandb: train/train_loss โ–ˆโ–‡โ–‡โ–†โ–†โ–…โ–…โ–„โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ– wandb: val/epoch โ–โ–โ–โ–‚โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆ wandb: val/hits@1 โ–โ–‚โ–ƒโ–ƒโ–„โ–„โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@10 โ–โ–‚โ–ƒโ–„โ–„โ–…โ–…โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@10_type โ–โ–ƒโ–„โ–„โ–…โ–…โ–†โ–†โ–†โ–†โ–‡โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@1_type โ–โ–„โ–„โ–‚โ–†โ–„โ–†โ–†โ–‡โ–†โ–†โ–„โ–†โ–‡โ–…โ–†โ–„โ–„โ–„โ–…โ–…โ–†โ–„โ–†โ–…โ–‡โ–†โ–‡โ–‡โ–ˆโ–‡ wandb: val/hits@3 โ–โ–‚โ–ƒโ–„โ–„โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@3_type โ–โ–„โ–„โ–„โ–…โ–…โ–†โ–†โ–‡โ–‡โ–†โ–†โ–†โ–‡โ–†โ–‡โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–‡โ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/mr โ–ˆโ–‡โ–†โ–…โ–„โ–„โ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ– wandb: val/mr_type โ–ˆโ–‡โ–†โ–†โ–†โ–†โ–…โ–…โ–…โ–„โ–„โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ– wandb: val/mrr โ–โ–‚โ–ƒโ–ƒโ–„โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/mrr_type โ–โ–ƒโ–„โ–„โ–…โ–…โ–†โ–†โ–‡โ–†โ–†โ–†โ–†โ–‡โ–†โ–‡โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–†โ–‡โ–‡โ–ˆโ–‡โ–ˆโ–ˆโ–ˆโ–ˆ wandb: wandb: Run summary: wandb: duration 9710.46408 wandb: test/hits@1 0.104 wandb: test/hits@10 0.304 wandb: test/hits@10_type 0.343 wandb: test/hits@1_type 0.128 wandb: test/hits@3 0.189 wandb: test/hits@3_type 0.217 wandb: test/mr 501.846 wandb: test/mr_type 272.098 wandb: test/mrr 0.172 wandb: test/mrr_type 0.2 wandb: train/epoch 3000 wandb: train/train_loss 0.9326 wandb: val/epoch 3099 wandb: val/hits@1 0.109 wandb: val/hits@10 0.308 wandb: val/hits@10_type 0.349 wandb: val/hits@1_type 0.134 wandb: val/hits@3 0.193 wandb: val/hits@3_type 0.222 wandb: val/mr 489.847 wandb: val/mr_type 265.74 wandb: val/mrr 0.176 wandb: val/mrr_type 0.206 wandb: wandb: ๐Ÿš€ View run peach-sweep-3 at: https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/7jqvkcn1 wandb: โญ๏ธ View project at: https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) wandb: Find logs at: ./wandb/run-20240522_151448-7jqvkcn1/logs DEBUG:connectionpool:2024-05-22 18:02:49:Resetting dropped connection: o151352.ingest.sentry.io DEBUG:connectionpool:2024-05-22 18:02:50:https://o151352.ingest.sentry.io:443 "POST /api/4504800232407040/envelope/ HTTP/1.1" 200 0