DEBUG:cmd:2024-13-05 14:11:06:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:cmd:2024-13-05 14:11:06:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-13-05 14:11:06:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-13-05 14:11:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:11:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: Currently logged in as: 3555028709. Use `wandb login --relogin` to force relogin DEBUG:connectionpool:2024-13-05 14:11:07:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-13-05 14:11:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:pyagent:2024-13-05 14:11:16:Starting sweep agent: entity=None, project=None, count=3 DEBUG:pyagent:2024-13-05 14:11:16:Agent._setup() DEBUG:pyagent:2024-13-05 14:11:16:Agent._register() DEBUG:connectionpool:2024-13-05 14:11:16:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-13-05 14:11:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:pyagent:2024-13-05 14:11:17:agent_id = QWdlbnQ6emFtcGNkYWQ= DEBUG:connectionpool:2024-13-05 14:11:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:pyagent:2024-13-05 14:11:17:Job received: Job(zvk2wjtb,{'adv_temperature': {'value': 3}, 'batch_size': {'value': 2048}, 'dataloader': {'value': 'KGEDataLoader'}, 'delta': {'value': 0.0001}, 'device': {'value': 'cuda:0'}, 'dim': {'value': 100}, 'ent_file': {'value': 'entity2id.txt'}, 'epochs': {'value': 10000}, 'in_path': {'value': '../../benchmarks/FB15K237/'}, 'l3_regul_rate': {'value': 0}, 'log_interval': {'value': 100}, 'loss': {'value': 'MarginLoss'}, 'lr': {'value': 0.6991266020888974}, 'lr_e': {'value': 0.4395081217606958}, 'margin': {'value': 3}, 'margin_e': {'value': 6}, 'metric': {'value': 'hits@10'}, 'model': {'value': 'TransR'}, 'neg_ent': {'value': 4}, 'norm_flag': {'value': True}, 'num_workers': {'value': 16}, 'opt_method': {'value': 'adagrad'}, 'opt_method_e': {'value': 'adam'}, 'p_norm': {'value': 1}, 'patience': {'value': 2}, 'prediction': {'value': 'all'}, 'rand_init': {'value': False}, 'regul_rate': {'value': 0}, 'rel_file': {'value': 'relation2id.txt'}, 'save_path': {'value': './model.pth'}, 'strategy': {'value': 'NegativeSampling'}, 'test_batch_size': {'value': 5}, 'test_file': {'value': 'test2id.txt'}, 'test_sampler': {'value': 'TradTestSampler'}, 'tester': {'value': 'Tester'}, 'train_file': {'value': 'train2id.txt'}, 'train_sampler': {'value': 'BernSampler'}, 'trainer': {'value': 'Trainer'}, 'type_constrain': {'value': True}, 'use_early_stopping': {'value': True}, 'use_gpu': {'value': True}, 'use_tqdm': {'value': False}, 'valid_file': {'value': 'valid2id.txt'}, 'valid_interval': {'value': 100}}) DEBUG:pyagent:2024-13-05 14:11:17:Spawning new thread for run zvk2wjtb. wandb: Agent Starting Run: zvk2wjtb with config: wandb: adv_temperature: 3 wandb: batch_size: 2048 wandb: dataloader: KGEDataLoader wandb: delta: 0.0001 wandb: device: cuda:0 wandb: dim: 100 wandb: ent_file: entity2id.txt wandb: epochs: 10000 wandb: in_path: ../../benchmarks/FB15K237/ wandb: l3_regul_rate: 0 wandb: log_interval: 100 wandb: loss: MarginLoss wandb: lr: 0.6991266020888974 wandb: lr_e: 0.4395081217606958 wandb: margin: 3 wandb: margin_e: 6 wandb: metric: hits@10 wandb: model: TransR wandb: neg_ent: 4 wandb: norm_flag: True wandb: num_workers: 16 wandb: opt_method: adagrad wandb: opt_method_e: adam wandb: p_norm: 1 wandb: patience: 2 wandb: prediction: all wandb: rand_init: False wandb: regul_rate: 0 wandb: rel_file: relation2id.txt wandb: save_path: ./model.pth wandb: strategy: NegativeSampling wandb: test_batch_size: 5 wandb: test_file: test2id.txt wandb: test_sampler: TradTestSampler wandb: tester: Tester wandb: train_file: train2id.txt wandb: train_sampler: BernSampler wandb: trainer: Trainer wandb: type_constrain: True wandb: use_early_stopping: True wandb: use_gpu: True wandb: use_tqdm: False wandb: valid_file: valid2id.txt wandb: valid_interval: 100 DEBUG:cmd:2024-13-05 14:11:17:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:cmd:2024-13-05 14:11:17:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-13-05 14:11:18:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-13-05 14:11:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:11:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:cmd:2024-13-05 14:11:19:Popen(['git', 'cat-file', '--batch-check'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=, shell=False, universal_newlines=False) wandb: - Waiting for wandb.init()... wandb: \ Waiting for wandb.init()... DEBUG:connectionpool:2024-13-05 14:11:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: Tracking run with wandb version 0.16.6 wandb: Run data is saved locally in /home/luyanfeng/my_code/github/pybind11-OpenKE/examples/TransR/wandb/run-20240513_141119-zvk2wjtb wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run hardy-sweep-1 wandb: โญ๏ธ View project at https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: ๐Ÿงน View sweep at https://wandb.ai/3555028709/pybind11-ke-sweeps/sweeps/bgb8gj47 wandb: ๐Ÿš€ View run at https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/zvk2wjtb DEBUG:connectionpool:2024-13-05 14:11:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:11:29:[cuda:0] Initialization completed, start model training. data_loader_config: {'batch_size': {'values': [512, 1024, 2048, 4096]}, 'dataloader': {'value': 'KGEDataLoader'}, 'ent_file': {'value': 'entity2id.txt'}, 'in_path': {'value': './'}, 'neg_ent': {'values': [1, 4, 16, 64]}, 'num_workers': {'value': 16}, 'rel_file': {'value': 'relation2id.txt'}, 'test_batch_size': {'value': 30}, 'test_file': {'value': 'test2id.txt'}, 'test_sampler': {'value': 'TradTestSampler'}, 'train_file': {'value': 'train2id.txt'}, 'train_sampler': {'value': 'BernSampler'}, 'type_constrain': {'value': True}, 'valid_file': {'value': 'valid2id.txt'}} kge_config: {'dim': {'values': [50, 100]}, 'lr_e': {'distribution': 'uniform', 'max': 1.0, 'min': 1e-05}, 'margin_e': {'values': [1.0, 3.0, 6.0]}, 'model': {'value': 'TransR'}, 'norm_flag': {'value': True}, 'opt_method_e': {'values': ['adam', 'adagrad', 'sgd']}, 'p_norm': {'values': [1, 2]}, 'rand_init': {'value': False}} loss_config: {'adv_temperature': {'values': [1.0, 3.0, 6.0]}, 'loss': {'value': 'MarginLoss'}, 'margin': {'values': [1.0, 3.0, 6.0]}} strategy_config: {'l3_regul_rate': {'value': 0.0}, 'regul_rate': {'value': 0.0}, 'strategy': {'value': 'NegativeSampling'}} tester_config: {'device': {'value': 'cuda:0'}, 'prediction': {'value': 'all'}, 'tester': {'value': 'Tester'}, 'use_gpu': {'value': True}, 'use_tqdm': {'value': False}} trainer_config: {'delta': {'value': 0.0001}, 'epochs': {'value': 10000}, 'log_interval': {'value': 100}, 'lr': {'distribution': 'uniform', 'max': 1.0, 'min': 1e-05}, 'metric': {'value': 'hits@10'}, 'opt_method': {'values': ['adam', 'adagrad', 'sgd']}, 'patience': {'value': 2}, 'save_path': {'value': './model.pth'}, 'trainer': {'value': 'Trainer'}, 'use_early_stopping': {'value': True}, 'valid_interval': {'value': 100}} sweep_config: {'method': 'bayes', 'metric': {'goal': 'maximize', 'name': 'val/hits@10'}, 'name': 'TransR_FB15K237', 'parameters': {'adv_temperature': {'values': [1.0, 3.0, 6.0]}, 'batch_size': {'values': [512, 1024, 2048, 4096]}, 'dataloader': {'value': 'KGEDataLoader'}, 'delta': {'value': 0.0001}, 'device': {'value': 'cuda:0'}, 'dim': {'values': [50, 100]}, 'ent_file': {'value': 'entity2id.txt'}, 'epochs': {'value': 10000}, 'in_path': {'value': '../../benchmarks/FB15K237/'}, 'l3_regul_rate': {'value': 0.0}, 'log_interval': {'value': 100}, 'loss': {'value': 'MarginLoss'}, 'lr': {'distribution': 'uniform', 'max': 1.0, 'min': 1e-05}, 'lr_e': {'distribution': 'uniform', 'max': 1.0, 'min': 1e-05}, 'margin': {'values': [1.0, 3.0, 6.0]}, 'margin_e': {'values': [1.0, 3.0, 6.0]}, 'metric': {'value': 'hits@10'}, 'model': {'value': 'TransR'}, 'neg_ent': {'values': [1, 4, 16, 64]}, 'norm_flag': {'value': True}, 'num_workers': {'value': 16}, 'opt_method': {'values': ['adam', 'adagrad', 'sgd']}, 'opt_method_e': {'values': ['adam', 'adagrad', 'sgd']}, 'p_norm': {'values': [1, 2]}, 'patience': {'value': 2}, 'prediction': {'value': 'all'}, 'rand_init': {'value': False}, 'regul_rate': {'value': 0.0}, 'rel_file': {'value': 'relation2id.txt'}, 'save_path': {'value': './model.pth'}, 'strategy': {'value': 'NegativeSampling'}, 'test_batch_size': {'value': 5}, 'test_file': {'value': 'test2id.txt'}, 'test_sampler': {'value': 'TradTestSampler'}, 'tester': {'value': 'Tester'}, 'train_file': {'value': 'train2id.txt'}, 'train_sampler': {'value': 'BernSampler'}, 'trainer': {'value': 'Trainer'}, 'type_constrain': {'value': True}, 'use_early_stopping': {'value': True}, 'use_gpu': {'value': True}, 'use_tqdm': {'value': False}, 'valid_file': {'value': 'valid2id.txt'}, 'valid_interval': {'value': 100}}} Create sweep with ID: bgb8gj47 Sweep URL: https://wandb.ai/3555028709/pybind11-ke-sweeps/sweeps/bgb8gj47 INFO:Trainer:2024-13-05 14:11:31:[cuda:0] The model training is completed, taking a total of 2.37873 seconds. INFO:Trainer:2024-13-05 14:11:32:[cuda:0] Initialization completed, start model training. DEBUG:connectionpool:2024-13-05 14:11:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:11:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:11:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:11:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:11:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:12:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:12:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:12:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:12:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:12:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:12:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:12:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:12:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:12:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:12:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:12:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:12:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:13:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:13:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:13:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:13:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:13:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:13:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:13:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:13:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:13:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:13:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:13:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:14:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:14:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:14:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:14:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:14:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:14:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:14:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:14:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:14:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:14:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:14:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:15:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:15:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:15:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:15:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:15:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:15:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:15:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:15:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:15:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:15:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:15:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:16:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:16:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:16:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:16:17:[cuda:0] Epoch 100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 14:16:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:16:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:16:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:16:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:16:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:16:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:16:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:16:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:17:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:17:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:17:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:17:12:mr: 378.509 INFO:Trainer:2024-13-05 14:17:12:mrr: 0.27 INFO:Trainer:2024-13-05 14:17:12:hits@1: 0.188 INFO:Trainer:2024-13-05 14:17:12:hits@3: 0.299 INFO:Trainer:2024-13-05 14:17:12:hits@10: 0.434 INFO:Trainer:2024-13-05 14:17:12:mr_type: 142.634 INFO:Trainer:2024-13-05 14:17:12:mrr_type: 0.302 INFO:Trainer:2024-13-05 14:17:12:hits@1_type: 0.216 INFO:Trainer:2024-13-05 14:17:12:hits@3_type: 0.33 INFO:Trainer:2024-13-05 14:17:12:hits@10_type: 0.473 INFO:EarlyStopping:2024-13-05 14:17:12:Validation score improved (-inf --> 0.434000). Saving model ... INFO:Trainer:2024-13-05 14:17:12:[cuda:0] Epoch [ 100/10000] | Batchsize: 2048 | loss: 4.482671 | 2.84404 seconds/epoch DEBUG:connectionpool:2024-13-05 14:17:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:17:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:17:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:17:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:17:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:17:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:17:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:17:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:18:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:18:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:18:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:18:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:18:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:18:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:18:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:18:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:18:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:18:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:18:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:18:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:19:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:19:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:19:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:19:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:19:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:19:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:19:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:19:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:19:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:19:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:19:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:20:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:20:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:20:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:20:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:20:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:20:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:20:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:20:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:20:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:20:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:20:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:21:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:21:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:21:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:21:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:21:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:21:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:21:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:21:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:21:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:21:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:21:52:[cuda:0] Epoch 200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 14:21:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:22:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:22:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:22:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:22:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:22:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:22:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:22:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:22:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:22:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:22:47:mr: 396.818 INFO:Trainer:2024-13-05 14:22:47:mrr: 0.281 INFO:Trainer:2024-13-05 14:22:47:hits@1: 0.197 INFO:Trainer:2024-13-05 14:22:47:hits@3: 0.312 INFO:Trainer:2024-13-05 14:22:47:hits@10: 0.448 INFO:Trainer:2024-13-05 14:22:47:mr_type: 133.81 INFO:Trainer:2024-13-05 14:22:47:mrr_type: 0.315 INFO:Trainer:2024-13-05 14:22:47:hits@1_type: 0.226 INFO:Trainer:2024-13-05 14:22:47:hits@3_type: 0.346 INFO:Trainer:2024-13-05 14:22:47:hits@10_type: 0.489 INFO:EarlyStopping:2024-13-05 14:22:47:Validation score improved (0.434000 --> 0.448000). Saving model ... INFO:Trainer:2024-13-05 14:22:47:[cuda:0] Epoch [ 200/10000] | Batchsize: 2048 | loss: 3.234025 | 3.09966 seconds/epoch DEBUG:connectionpool:2024-13-05 14:22:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:22:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:23:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:23:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:23:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:23:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:23:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:23:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:23:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:23:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:23:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:23:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:23:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:24:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:24:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:24:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:24:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:24:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:24:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:24:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:24:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:24:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:24:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:24:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:25:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:25:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:25:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:25:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:25:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:25:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:25:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:25:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:25:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:25:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:25:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:26:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:26:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:26:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:26:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:26:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:26:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:26:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:26:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:26:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:26:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:26:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:27:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:27:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:27:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:27:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:27:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:27:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:27:31:[cuda:0] Epoch 300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 14:27:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:27:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:27:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:27:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:27:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:28:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:28:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:28:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:28:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:28:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:28:26:mr: 423.176 INFO:Trainer:2024-13-05 14:28:26:mrr: 0.282 INFO:Trainer:2024-13-05 14:28:26:hits@1: 0.197 INFO:Trainer:2024-13-05 14:28:26:hits@3: 0.312 INFO:Trainer:2024-13-05 14:28:26:hits@10: 0.45 INFO:Trainer:2024-13-05 14:28:26:mr_type: 131.168 INFO:Trainer:2024-13-05 14:28:26:mrr_type: 0.317 INFO:Trainer:2024-13-05 14:28:26:hits@1_type: 0.228 INFO:Trainer:2024-13-05 14:28:26:hits@3_type: 0.347 INFO:Trainer:2024-13-05 14:28:26:hits@10_type: 0.494 INFO:EarlyStopping:2024-13-05 14:28:26:Validation score improved (0.448000 --> 0.450000). Saving model ... INFO:Trainer:2024-13-05 14:28:26:[cuda:0] Epoch [ 300/10000] | Batchsize: 2048 | loss: 2.684743 | 3.19442 seconds/epoch DEBUG:connectionpool:2024-13-05 14:28:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:28:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:28:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:28:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:28:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:28:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:29:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:29:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:29:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:29:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:29:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:29:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:29:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:29:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:29:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:29:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:29:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:30:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:30:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:30:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:30:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:30:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:30:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:30:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:30:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:30:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:30:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:30:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:31:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:31:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:31:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:31:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:31:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:31:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:31:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:31:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:31:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:31:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:31:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:31:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:32:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:32:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:32:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:32:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:32:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:32:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:32:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:32:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:32:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:32:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:32:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:33:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:33:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:33:11:[cuda:0] Epoch 400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 14:33:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:33:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:33:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:33:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:33:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:33:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:33:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:33:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:33:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:34:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:34:07:mr: 439.162 INFO:Trainer:2024-13-05 14:34:07:mrr: 0.285 INFO:Trainer:2024-13-05 14:34:07:hits@1: 0.201 INFO:Trainer:2024-13-05 14:34:07:hits@3: 0.316 INFO:Trainer:2024-13-05 14:34:07:hits@10: 0.454 INFO:Trainer:2024-13-05 14:34:07:mr_type: 128.856 INFO:Trainer:2024-13-05 14:34:07:mrr_type: 0.321 INFO:Trainer:2024-13-05 14:34:07:hits@1_type: 0.232 INFO:Trainer:2024-13-05 14:34:07:hits@3_type: 0.352 INFO:Trainer:2024-13-05 14:34:07:hits@10_type: 0.499 INFO:EarlyStopping:2024-13-05 14:34:07:Validation score improved (0.450000 --> 0.454000). Saving model ... INFO:Trainer:2024-13-05 14:34:07:[cuda:0] Epoch [ 400/10000] | Batchsize: 2048 | loss: 2.278421 | 3.24804 seconds/epoch DEBUG:connectionpool:2024-13-05 14:34:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:34:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:34:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:34:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:34:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:34:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:34:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:34:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:34:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:34:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:35:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:35:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:35:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:35:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:35:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:35:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:35:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:35:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:35:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:35:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:35:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:36:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:36:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:36:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:36:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:36:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:36:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:36:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:36:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:36:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:36:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:36:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:36:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:37:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:37:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:37:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:37:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:37:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:37:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:37:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:37:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:37:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:37:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:37:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:38:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:38:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:38:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:38:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:38:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:38:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:38:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:38:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:38:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:38:49:[cuda:0] Epoch 500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 14:38:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:38:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:39:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:39:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:39:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:39:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:39:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:39:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:39:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:39:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:39:45:mr: 454.621 INFO:Trainer:2024-13-05 14:39:45:mrr: 0.287 INFO:Trainer:2024-13-05 14:39:45:hits@1: 0.202 INFO:Trainer:2024-13-05 14:39:45:hits@3: 0.319 INFO:Trainer:2024-13-05 14:39:45:hits@10: 0.455 INFO:Trainer:2024-13-05 14:39:45:mr_type: 127.716 INFO:Trainer:2024-13-05 14:39:45:mrr_type: 0.324 INFO:Trainer:2024-13-05 14:39:45:hits@1_type: 0.234 INFO:Trainer:2024-13-05 14:39:45:hits@3_type: 0.357 INFO:Trainer:2024-13-05 14:39:45:hits@10_type: 0.503 INFO:EarlyStopping:2024-13-05 14:39:45:Validation score improved (0.454000 --> 0.455000). Saving model ... INFO:Trainer:2024-13-05 14:39:45:[cuda:0] Epoch [ 500/10000] | Batchsize: 2048 | loss: 2.084943 | 3.27418 seconds/epoch DEBUG:connectionpool:2024-13-05 14:39:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:39:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:39:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:40:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:40:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:40:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:40:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:40:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:40:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:40:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:40:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:40:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:40:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:40:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:41:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:41:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:41:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:41:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:41:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:41:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:41:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:41:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:41:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:41:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:41:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:42:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:42:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:42:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:42:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:42:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:42:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:42:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:42:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:42:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:42:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:42:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:43:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:43:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:43:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:43:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:43:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:43:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:43:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:43:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:43:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:43:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:43:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:43:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:44:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:44:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:44:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:44:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:44:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:44:27:[cuda:0] Epoch 600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 14:44:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:44:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:44:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:44:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:44:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:44:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:45:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:45:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:45:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:45:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:45:22:mr: 465.254 INFO:Trainer:2024-13-05 14:45:22:mrr: 0.288 INFO:Trainer:2024-13-05 14:45:22:hits@1: 0.202 INFO:Trainer:2024-13-05 14:45:22:hits@3: 0.322 INFO:Trainer:2024-13-05 14:45:22:hits@10: 0.457 INFO:Trainer:2024-13-05 14:45:22:mr_type: 126.458 INFO:Trainer:2024-13-05 14:45:22:mrr_type: 0.325 INFO:Trainer:2024-13-05 14:45:22:hits@1_type: 0.234 INFO:Trainer:2024-13-05 14:45:22:hits@3_type: 0.358 INFO:Trainer:2024-13-05 14:45:22:hits@10_type: 0.505 INFO:EarlyStopping:2024-13-05 14:45:22:Validation score improved (0.455000 --> 0.457000). Saving model ... INFO:Trainer:2024-13-05 14:45:22:[cuda:0] Epoch [ 600/10000] | Batchsize: 2048 | loss: 1.900912 | 3.29085 seconds/epoch DEBUG:connectionpool:2024-13-05 14:45:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:45:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:45:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:45:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:45:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:45:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:45:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:46:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:46:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:46:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:46:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:46:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:46:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:46:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:46:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:46:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:46:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:46:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:47:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:47:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:47:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:47:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:47:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:47:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:47:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:47:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:47:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:47:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:47:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:48:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:48:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:48:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:48:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:48:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:48:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:48:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:48:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:48:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:48:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:48:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:49:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:49:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:49:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:49:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:49:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:49:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:49:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:49:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:49:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:49:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:49:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:49:59:[cuda:0] Epoch 700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 14:50:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:50:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:50:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:50:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:50:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:50:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:50:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:50:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:50:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:50:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:50:55:mr: 473.898 INFO:Trainer:2024-13-05 14:50:55:mrr: 0.287 INFO:Trainer:2024-13-05 14:50:55:hits@1: 0.202 INFO:Trainer:2024-13-05 14:50:55:hits@3: 0.319 INFO:Trainer:2024-13-05 14:50:55:hits@10: 0.455 INFO:Trainer:2024-13-05 14:50:55:mr_type: 125.686 INFO:Trainer:2024-13-05 14:50:55:mrr_type: 0.326 INFO:Trainer:2024-13-05 14:50:55:hits@1_type: 0.235 INFO:Trainer:2024-13-05 14:50:55:hits@3_type: 0.357 INFO:Trainer:2024-13-05 14:50:55:hits@10_type: 0.506 INFO:EarlyStopping:2024-13-05 14:50:55:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-13-05 14:50:55:[cuda:0] Epoch [ 700/10000] | Batchsize: 2048 | loss: 1.773239 | 3.29586 seconds/epoch DEBUG:connectionpool:2024-13-05 14:50:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:51:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:51:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:51:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:51:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:51:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:51:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:51:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:51:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:51:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:51:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:51:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:52:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:52:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:52:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:52:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:52:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:52:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:52:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:52:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:52:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:52:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:52:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:53:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:53:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:53:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:53:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:53:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:53:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:53:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:53:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:53:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:53:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:53:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:54:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:54:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:54:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:54:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:54:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:54:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:54:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:54:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:54:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:54:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:54:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:55:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:55:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:55:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:55:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:55:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:55:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:55:29:[cuda:0] Epoch 800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 14:55:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:55:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:55:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:55:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:55:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:56:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:56:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:56:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:56:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:56:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:56:25:mr: 483.161 INFO:Trainer:2024-13-05 14:56:25:mrr: 0.286 INFO:Trainer:2024-13-05 14:56:25:hits@1: 0.201 INFO:Trainer:2024-13-05 14:56:25:hits@3: 0.318 INFO:Trainer:2024-13-05 14:56:25:hits@10: 0.453 INFO:Trainer:2024-13-05 14:56:25:mr_type: 125.525 INFO:Trainer:2024-13-05 14:56:25:mrr_type: 0.326 INFO:Trainer:2024-13-05 14:56:25:hits@1_type: 0.235 INFO:Trainer:2024-13-05 14:56:25:hits@3_type: 0.359 INFO:Trainer:2024-13-05 14:56:25:hits@10_type: 0.506 INFO:EarlyStopping:2024-13-05 14:56:25:EarlyStopping counter: 2 / 2 INFO:Trainer:2024-13-05 14:56:25:[cuda:0] Send an early stopping signal INFO:Trainer:2024-13-05 14:56:25:[cuda:0] The model training is completed, taking a total of 2636.98764 seconds. INFO:Trainer:2024-13-05 14:56:25:[cuda:0] Model saved at ./model.pth. INFO:Trainer:2024-13-05 14:56:25:[cuda:0] The model starts evaluating in the test set. DEBUG:connectionpool:2024-13-05 14:56:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:56:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:56:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:56:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:56:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:56:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:57:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:57:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:57:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:57:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:57:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:57:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:57:29:mr: 492.518 INFO:Trainer:2024-13-05 14:57:29:mrr: 0.283 INFO:Trainer:2024-13-05 14:57:29:hits@1: 0.198 INFO:Trainer:2024-13-05 14:57:29:hits@3: 0.315 INFO:Trainer:2024-13-05 14:57:29:hits@10: 0.451 INFO:Trainer:2024-13-05 14:57:29:mr_type: 129.315 INFO:Trainer:2024-13-05 14:57:29:mrr_type: 0.322 INFO:Trainer:2024-13-05 14:57:29:hits@1_type: 0.231 INFO:Trainer:2024-13-05 14:57:29:hits@3_type: 0.354 INFO:Trainer:2024-13-05 14:57:29:hits@10_type: 0.501 wandb: - 0.014 MB of 0.014 MB uploaded wandb: \ 0.014 MB of 0.081 MB uploaded DEBUG:connectionpool:2024-13-05 14:57:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: | 0.074 MB of 0.081 MB uploaded wandb: / 0.074 MB of 0.081 MB uploaded wandb: - 0.074 MB of 0.081 MB uploaded wandb: \ 0.081 MB of 0.081 MB uploaded wandb: wandb: wandb: Run history: wandb: duration โ– wandb: test/hits@1 โ– wandb: test/hits@10 โ– wandb: test/hits@10_type โ– wandb: test/hits@1_type โ– wandb: test/hits@3 โ– wandb: test/hits@3_type โ– wandb: test/mr โ– wandb: test/mr_type โ– wandb: test/mrr โ– wandb: test/mrr_type โ– wandb: train/epoch โ–โ–‚โ–ƒโ–…โ–†โ–‡โ–ˆ wandb: train/train_loss โ–ˆโ–…โ–ƒโ–‚โ–‚โ–โ– wandb: val/epoch โ–โ–‚โ–ƒโ–„โ–…โ–†โ–‡โ–ˆ wandb: val/hits@1 โ–โ–…โ–…โ–‡โ–ˆโ–ˆโ–ˆโ–‡ wandb: val/hits@10 โ–โ–…โ–†โ–‡โ–‡โ–ˆโ–‡โ–‡ wandb: val/hits@10_type โ–โ–„โ–…โ–‡โ–‡โ–ˆโ–ˆโ–ˆ wandb: val/hits@1_type โ–โ–…โ–…โ–‡โ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@3 โ–โ–…โ–…โ–†โ–‡โ–ˆโ–‡โ–‡ wandb: val/hits@3_type โ–โ–…โ–…โ–†โ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/mr โ–โ–‚โ–„โ–…โ–†โ–‡โ–‡โ–ˆ wandb: val/mr_type โ–ˆโ–„โ–ƒโ–‚โ–‚โ–โ–โ– wandb: val/mrr โ–โ–…โ–†โ–‡โ–ˆโ–ˆโ–ˆโ–‡ wandb: val/mrr_type โ–โ–…โ–…โ–‡โ–‡โ–ˆโ–ˆโ–ˆ wandb: wandb: Run summary: wandb: duration 2636.98764 wandb: test/hits@1 0.198 wandb: test/hits@10 0.451 wandb: test/hits@10_type 0.501 wandb: test/hits@1_type 0.231 wandb: test/hits@3 0.315 wandb: test/hits@3_type 0.354 wandb: test/mr 492.518 wandb: test/mr_type 129.315 wandb: test/mrr 0.283 wandb: test/mrr_type 0.322 wandb: train/epoch 700 wandb: train/train_loss 1.77324 wandb: val/epoch 799 wandb: val/hits@1 0.201 wandb: val/hits@10 0.453 wandb: val/hits@10_type 0.506 wandb: val/hits@1_type 0.235 wandb: val/hits@3 0.318 wandb: val/hits@3_type 0.359 wandb: val/mr 483.161 wandb: val/mr_type 125.525 wandb: val/mrr 0.286 wandb: val/mrr_type 0.326 wandb: wandb: ๐Ÿš€ View run hardy-sweep-1 at: https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/zvk2wjtb wandb: โญ๏ธ View project at: https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) wandb: Find logs at: ./wandb/run-20240513_141119-zvk2wjtb/logs DEBUG:connectionpool:2024-13-05 14:57:37:Starting new HTTPS connection (1): o151352.ingest.sentry.io:443 DEBUG:connectionpool:2024-13-05 14:57:38:https://o151352.ingest.sentry.io:443 "POST /api/4504800232407040/envelope/ HTTP/1.1" 200 0 DEBUG:connectionpool:2024-13-05 14:57:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: Agent Starting Run: yjp30i5f with config: wandb: adv_temperature: 1 wandb: batch_size: 2048 wandb: dataloader: KGEDataLoader wandb: delta: 0.0001 wandb: device: cuda:0 wandb: dim: 50 wandb: ent_file: entity2id.txt wandb: epochs: 10000 wandb: in_path: ../../benchmarks/FB15K237/ wandb: l3_regul_rate: 0 wandb: log_interval: 100 wandb: loss: MarginLoss wandb: lr: 0.4424616001116982 wandb: lr_e: 0.5109990180285069 wandb: margin: 1 wandb: margin_e: 3 wandb: metric: hits@10 wandb: model: TransR wandb: neg_ent: 1 wandb: norm_flag: True wandb: num_workers: 16 wandb: opt_method: adam wandb: opt_method_e: adagrad wandb: p_norm: 1 wandb: patience: 2 wandb: prediction: all wandb: rand_init: False wandb: regul_rate: 0 wandb: rel_file: relation2id.txt wandb: save_path: ./model.pth wandb: strategy: NegativeSampling wandb: test_batch_size: 5 wandb: test_file: test2id.txt wandb: test_sampler: TradTestSampler wandb: tester: Tester wandb: train_file: train2id.txt wandb: train_sampler: BernSampler wandb: trainer: Trainer wandb: type_constrain: True wandb: use_early_stopping: True wandb: use_gpu: True wandb: use_tqdm: False wandb: valid_file: valid2id.txt wandb: valid_interval: 100 DEBUG:cmd:2024-13-05 14:57:41:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:cmd:2024-13-05 14:57:41:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-13-05 14:57:42:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-13-05 14:57:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:57:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:cmd:2024-13-05 14:57:42:Popen(['git', 'cat-file', '--batch-check'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=, shell=False, universal_newlines=False) wandb: - Waiting for wandb.init()... wandb: \ Waiting for wandb.init()... wandb: wandb version 0.17.0 is available! To upgrade, please run: wandb: $ pip install wandb --upgrade wandb: Tracking run with wandb version 0.16.6 wandb: Run data is saved locally in /home/luyanfeng/my_code/github/pybind11-OpenKE/examples/TransR/wandb/run-20240513_145742-yjp30i5f wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run stilted-sweep-2 wandb: โญ๏ธ View project at https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: ๐Ÿงน View sweep at https://wandb.ai/3555028709/pybind11-ke-sweeps/sweeps/bgb8gj47 wandb: ๐Ÿš€ View run at https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/yjp30i5f DEBUG:connectionpool:2024-13-05 14:57:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:57:48:[cuda:0] Initialization completed, start model training. INFO:Trainer:2024-13-05 14:57:50:[cuda:0] The model training is completed, taking a total of 2.26587 seconds. DEBUG:connectionpool:2024-13-05 14:57:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 14:57:51:[cuda:0] Initialization completed, start model training. DEBUG:connectionpool:2024-13-05 14:57:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:58:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:58:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:58:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:58:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:58:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:58:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:58:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:58:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:58:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:58:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:58:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:59:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:59:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:59:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:59:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:59:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:59:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:59:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:59:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:59:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:59:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 14:59:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:00:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:00:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:00:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:00:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:00:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:00:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:00:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:00:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:00:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:00:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:00:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:00:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:01:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:01:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:01:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:01:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:01:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:01:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:01:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:01:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:01:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:01:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:01:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:02:01:[cuda:0] Epoch 100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 15:02:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:02:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:02:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:02:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:02:24:mr: 426.253 INFO:Trainer:2024-13-05 15:02:24:mrr: 0.172 INFO:Trainer:2024-13-05 15:02:24:hits@1: 0.099 INFO:Trainer:2024-13-05 15:02:24:hits@3: 0.191 INFO:Trainer:2024-13-05 15:02:24:hits@10: 0.314 INFO:Trainer:2024-13-05 15:02:24:mr_type: 179.431 INFO:Trainer:2024-13-05 15:02:24:mrr_type: 0.221 INFO:Trainer:2024-13-05 15:02:24:hits@1_type: 0.144 INFO:Trainer:2024-13-05 15:02:24:hits@3_type: 0.238 INFO:Trainer:2024-13-05 15:02:24:hits@10_type: 0.373 INFO:EarlyStopping:2024-13-05 15:02:24:Validation score improved (-inf --> 0.314000). Saving model ... INFO:Trainer:2024-13-05 15:02:24:[cuda:0] Epoch [ 100/10000] | Batchsize: 2048 | loss: 1.347817 | 2.49696 seconds/epoch DEBUG:connectionpool:2024-13-05 15:02:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:02:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:02:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:02:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:02:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:02:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:02:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:03:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:03:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:03:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:03:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:03:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:03:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:03:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:03:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:03:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:03:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:03:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:04:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:04:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:04:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:04:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:04:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:04:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:04:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:04:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:04:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:04:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:04:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:05:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:05:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:05:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:05:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:05:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:05:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:05:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:05:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:05:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:05:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:05:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:06:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:06:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:06:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:06:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:06:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:06:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:06:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:06:34:[cuda:0] Epoch 200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 15:06:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:06:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:06:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:06:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:06:58:mr: 350.04 INFO:Trainer:2024-13-05 15:06:58:mrr: 0.214 INFO:Trainer:2024-13-05 15:06:58:hits@1: 0.136 INFO:Trainer:2024-13-05 15:06:58:hits@3: 0.235 INFO:Trainer:2024-13-05 15:06:58:hits@10: 0.369 INFO:Trainer:2024-13-05 15:06:58:mr_type: 150.453 INFO:Trainer:2024-13-05 15:06:58:mrr_type: 0.255 INFO:Trainer:2024-13-05 15:06:58:hits@1_type: 0.171 INFO:Trainer:2024-13-05 15:06:58:hits@3_type: 0.277 INFO:Trainer:2024-13-05 15:06:58:hits@10_type: 0.423 INFO:EarlyStopping:2024-13-05 15:06:58:Validation score improved (0.314000 --> 0.369000). Saving model ... INFO:Trainer:2024-13-05 15:06:58:[cuda:0] Epoch [ 200/10000] | Batchsize: 2048 | loss: 0.713372 | 2.61623 seconds/epoch DEBUG:connectionpool:2024-13-05 15:06:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:07:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:07:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:07:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:07:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:07:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:07:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:07:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:07:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:07:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:07:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:07:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:08:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:08:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:08:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:08:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:08:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:08:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:08:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:08:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:08:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:08:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:08:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:09:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:09:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:09:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:09:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:09:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:09:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:09:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:09:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:09:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:09:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:09:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:10:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:10:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:10:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:10:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:10:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:10:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:10:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:10:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:10:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:10:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:10:56:[cuda:0] Epoch 300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 15:10:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:11:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:11:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:11:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:11:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:11:21:mr: 342.22 INFO:Trainer:2024-13-05 15:11:21:mrr: 0.23 INFO:Trainer:2024-13-05 15:11:21:hits@1: 0.15 INFO:Trainer:2024-13-05 15:11:21:hits@3: 0.253 INFO:Trainer:2024-13-05 15:11:21:hits@10: 0.39 INFO:Trainer:2024-13-05 15:11:21:mr_type: 139.89 INFO:Trainer:2024-13-05 15:11:21:mrr_type: 0.271 INFO:Trainer:2024-13-05 15:11:21:hits@1_type: 0.186 INFO:Trainer:2024-13-05 15:11:21:hits@3_type: 0.295 INFO:Trainer:2024-13-05 15:11:21:hits@10_type: 0.441 INFO:EarlyStopping:2024-13-05 15:11:21:Validation score improved (0.369000 --> 0.390000). Saving model ... INFO:Trainer:2024-13-05 15:11:21:[cuda:0] Epoch [ 300/10000] | Batchsize: 2048 | loss: 0.528804 | 2.61482 seconds/epoch DEBUG:connectionpool:2024-13-05 15:11:24:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-13-05 15:11:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:11:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:11:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:11:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:11:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:11:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:11:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:12:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:12:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:12:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:12:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:12:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:12:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:12:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:12:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:12:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:12:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:12:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:13:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:13:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:13:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:13:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:13:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:13:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:13:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:13:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:13:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:13:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:13:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:14:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:14:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:14:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:14:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:14:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:14:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:14:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:14:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:14:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:14:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:14:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:15:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:15:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:15:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:15:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:15:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:15:26:[cuda:0] Epoch 400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 15:15:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:15:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:15:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:15:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:15:50:mr: 321.982 INFO:Trainer:2024-13-05 15:15:50:mrr: 0.239 INFO:Trainer:2024-13-05 15:15:50:hits@1: 0.158 INFO:Trainer:2024-13-05 15:15:50:hits@3: 0.263 INFO:Trainer:2024-13-05 15:15:50:hits@10: 0.402 INFO:Trainer:2024-13-05 15:15:50:mr_type: 134.335 INFO:Trainer:2024-13-05 15:15:50:mrr_type: 0.277 INFO:Trainer:2024-13-05 15:15:50:hits@1_type: 0.189 INFO:Trainer:2024-13-05 15:15:50:hits@3_type: 0.304 INFO:Trainer:2024-13-05 15:15:50:hits@10_type: 0.454 INFO:EarlyStopping:2024-13-05 15:15:50:Validation score improved (0.390000 --> 0.402000). Saving model ... INFO:Trainer:2024-13-05 15:15:50:[cuda:0] Epoch [ 400/10000] | Batchsize: 2048 | loss: 0.421434 | 2.63822 seconds/epoch DEBUG:connectionpool:2024-13-05 15:15:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:15:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:16:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:16:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:16:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:16:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:16:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:16:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:16:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:16:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:16:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:16:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:16:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:17:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:17:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:17:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:17:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:17:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:17:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:17:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:17:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:17:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:17:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:17:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:18:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:18:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:18:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:18:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:18:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:18:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:18:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:18:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:18:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:18:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:18:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:19:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:19:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:19:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:19:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:19:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:19:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:19:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:19:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:19:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:19:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:19:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:19:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:19:59:[cuda:0] Epoch 500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 15:20:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:20:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:20:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:20:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:20:23:mr: 310.776 INFO:Trainer:2024-13-05 15:20:23:mrr: 0.247 INFO:Trainer:2024-13-05 15:20:23:hits@1: 0.163 INFO:Trainer:2024-13-05 15:20:23:hits@3: 0.273 INFO:Trainer:2024-13-05 15:20:23:hits@10: 0.412 INFO:Trainer:2024-13-05 15:20:23:mr_type: 128.413 INFO:Trainer:2024-13-05 15:20:23:mrr_type: 0.287 INFO:Trainer:2024-13-05 15:20:23:hits@1_type: 0.197 INFO:Trainer:2024-13-05 15:20:23:hits@3_type: 0.314 INFO:Trainer:2024-13-05 15:20:23:hits@10_type: 0.465 INFO:EarlyStopping:2024-13-05 15:20:23:Validation score improved (0.402000 --> 0.412000). Saving model ... INFO:Trainer:2024-13-05 15:20:23:[cuda:0] Epoch [ 500/10000] | Batchsize: 2048 | loss: 0.348365 | 2.65501 seconds/epoch DEBUG:connectionpool:2024-13-05 15:20:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:20:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:20:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:20:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:20:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:20:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:20:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:21:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:21:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:21:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:21:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:21:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:21:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:21:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:21:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:21:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:21:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:21:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:22:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:22:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:22:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:22:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:22:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:22:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:22:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:22:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:22:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:22:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:23:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:23:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:23:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:23:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:23:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:23:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:23:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:23:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:23:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:23:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:23:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:24:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:24:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:24:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:24:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:24:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:24:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:24:27:[cuda:0] Epoch 600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 15:24:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:24:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:24:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:24:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:24:51:mr: 314.769 INFO:Trainer:2024-13-05 15:24:51:mrr: 0.255 INFO:Trainer:2024-13-05 15:24:51:hits@1: 0.173 INFO:Trainer:2024-13-05 15:24:51:hits@3: 0.28 INFO:Trainer:2024-13-05 15:24:51:hits@10: 0.42 INFO:Trainer:2024-13-05 15:24:51:mr_type: 126.322 INFO:Trainer:2024-13-05 15:24:51:mrr_type: 0.293 INFO:Trainer:2024-13-05 15:24:51:hits@1_type: 0.205 INFO:Trainer:2024-13-05 15:24:51:hits@3_type: 0.32 INFO:Trainer:2024-13-05 15:24:51:hits@10_type: 0.469 INFO:EarlyStopping:2024-13-05 15:24:51:Validation score improved (0.412000 --> 0.420000). Saving model ... INFO:Trainer:2024-13-05 15:24:51:[cuda:0] Epoch [ 600/10000] | Batchsize: 2048 | loss: 0.326364 | 2.65917 seconds/epoch DEBUG:connectionpool:2024-13-05 15:24:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:24:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:25:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:25:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:25:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:25:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:25:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:25:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:25:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:25:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:25:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:25:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:25:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:26:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:26:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:26:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:26:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:26:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:26:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:26:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:26:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:26:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:26:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:26:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:27:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:27:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:27:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:27:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:27:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:27:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:27:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:27:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:27:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:27:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:27:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:27:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:28:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:28:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:28:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:28:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:28:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:28:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:28:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:28:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:28:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:28:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:28:55:[cuda:0] Epoch 700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 15:28:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:29:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:29:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:29:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:29:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:29:21:mr: 316.366 INFO:Trainer:2024-13-05 15:29:21:mrr: 0.259 INFO:Trainer:2024-13-05 15:29:21:hits@1: 0.177 INFO:Trainer:2024-13-05 15:29:21:hits@3: 0.286 INFO:Trainer:2024-13-05 15:29:21:hits@10: 0.422 INFO:Trainer:2024-13-05 15:29:21:mr_type: 125.713 INFO:Trainer:2024-13-05 15:29:21:mrr_type: 0.298 INFO:Trainer:2024-13-05 15:29:21:hits@1_type: 0.209 INFO:Trainer:2024-13-05 15:29:21:hits@3_type: 0.329 INFO:Trainer:2024-13-05 15:29:21:hits@10_type: 0.475 INFO:EarlyStopping:2024-13-05 15:29:21:Validation score improved (0.420000 --> 0.422000). Saving model ... INFO:Trainer:2024-13-05 15:29:21:[cuda:0] Epoch [ 700/10000] | Batchsize: 2048 | loss: 0.307767 | 2.66272 seconds/epoch DEBUG:connectionpool:2024-13-05 15:29:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:29:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:29:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:29:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:29:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:29:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:29:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:30:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:30:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:30:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:30:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:30:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:30:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:30:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:30:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:30:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:30:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:30:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:31:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:31:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:31:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:31:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:31:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:31:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:31:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:31:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:31:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:31:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:31:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:32:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:32:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:32:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:32:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:32:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:32:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:32:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:32:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:32:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:32:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:32:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:33:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:33:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:33:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:33:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:33:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:33:22:[cuda:0] Epoch 800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 15:33:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:33:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:33:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:33:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:33:48:mr: 317.262 INFO:Trainer:2024-13-05 15:33:48:mrr: 0.259 INFO:Trainer:2024-13-05 15:33:48:hits@1: 0.176 INFO:Trainer:2024-13-05 15:33:48:hits@3: 0.285 INFO:Trainer:2024-13-05 15:33:48:hits@10: 0.424 INFO:Trainer:2024-13-05 15:33:48:mr_type: 124.206 INFO:Trainer:2024-13-05 15:33:48:mrr_type: 0.298 INFO:Trainer:2024-13-05 15:33:48:hits@1_type: 0.209 INFO:Trainer:2024-13-05 15:33:48:hits@3_type: 0.325 INFO:Trainer:2024-13-05 15:33:48:hits@10_type: 0.475 INFO:EarlyStopping:2024-13-05 15:33:48:Validation score improved (0.422000 --> 0.424000). Saving model ... INFO:Trainer:2024-13-05 15:33:48:[cuda:0] Epoch [ 800/10000] | Batchsize: 2048 | loss: 0.257680 | 2.66384 seconds/epoch DEBUG:connectionpool:2024-13-05 15:33:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:33:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:33:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:34:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:34:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:34:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:34:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:34:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:34:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:34:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:34:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:34:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:34:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:34:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:35:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:35:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:35:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:35:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:35:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:35:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:35:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:35:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:35:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:35:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:35:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:36:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:36:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:36:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:36:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:36:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:36:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:36:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:36:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:36:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:36:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:36:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:37:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:37:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:37:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:37:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:37:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:37:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:37:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:37:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:37:45:[cuda:0] Epoch 900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 15:37:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:37:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:37:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:38:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:38:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:38:10:mr: 311.618 INFO:Trainer:2024-13-05 15:38:10:mrr: 0.264 INFO:Trainer:2024-13-05 15:38:10:hits@1: 0.181 INFO:Trainer:2024-13-05 15:38:10:hits@3: 0.293 INFO:Trainer:2024-13-05 15:38:10:hits@10: 0.431 INFO:Trainer:2024-13-05 15:38:10:mr_type: 121.934 INFO:Trainer:2024-13-05 15:38:10:mrr_type: 0.303 INFO:Trainer:2024-13-05 15:38:10:hits@1_type: 0.213 INFO:Trainer:2024-13-05 15:38:10:hits@3_type: 0.333 INFO:Trainer:2024-13-05 15:38:10:hits@10_type: 0.483 INFO:EarlyStopping:2024-13-05 15:38:10:Validation score improved (0.424000 --> 0.431000). Saving model ... INFO:Trainer:2024-13-05 15:38:10:[cuda:0] Epoch [ 900/10000] | Batchsize: 2048 | loss: 0.242212 | 2.66015 seconds/epoch DEBUG:connectionpool:2024-13-05 15:38:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:38:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:38:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:38:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:38:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:38:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:38:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:38:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:38:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:39:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:39:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:39:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:39:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:39:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:39:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:39:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:39:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:39:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:39:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:39:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:40:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:40:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:40:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:40:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:40:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:40:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:40:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:40:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:40:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:40:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:40:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:41:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:41:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:41:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:41:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:41:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:41:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:41:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:41:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:41:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:41:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:41:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:42:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:42:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:42:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:42:17:[cuda:0] Epoch 1000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 15:42:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:42:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:42:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:42:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:42:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:42:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:42:46:mr: 318.577 INFO:Trainer:2024-13-05 15:42:46:mrr: 0.266 INFO:Trainer:2024-13-05 15:42:46:hits@1: 0.184 INFO:Trainer:2024-13-05 15:42:46:hits@3: 0.293 INFO:Trainer:2024-13-05 15:42:46:hits@10: 0.43 INFO:Trainer:2024-13-05 15:42:46:mr_type: 121.748 INFO:Trainer:2024-13-05 15:42:46:mrr_type: 0.304 INFO:Trainer:2024-13-05 15:42:46:hits@1_type: 0.215 INFO:Trainer:2024-13-05 15:42:46:hits@3_type: 0.333 INFO:Trainer:2024-13-05 15:42:46:hits@10_type: 0.482 INFO:EarlyStopping:2024-13-05 15:42:46:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-13-05 15:42:46:[cuda:0] Epoch [1000/10000] | Batchsize: 2048 | loss: 0.249729 | 2.66622 seconds/epoch DEBUG:connectionpool:2024-13-05 15:42:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:42:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:43:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:43:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:43:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:43:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:43:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:43:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:43:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:43:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:43:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:43:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:43:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:44:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:44:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:44:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:44:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:44:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:44:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:44:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:44:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:44:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:44:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:44:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:44:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:45:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:45:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:45:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:45:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:45:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:45:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:45:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:45:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:45:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:45:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:45:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:46:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:46:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:46:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:46:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:46:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:46:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:46:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:46:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:46:49:[cuda:0] Epoch 1100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 15:46:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:46:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:47:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:47:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:47:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:47:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:47:18:mr: 321.028 INFO:Trainer:2024-13-05 15:47:18:mrr: 0.265 INFO:Trainer:2024-13-05 15:47:18:hits@1: 0.184 INFO:Trainer:2024-13-05 15:47:18:hits@3: 0.289 INFO:Trainer:2024-13-05 15:47:18:hits@10: 0.432 INFO:Trainer:2024-13-05 15:47:18:mr_type: 119.844 INFO:Trainer:2024-13-05 15:47:18:mrr_type: 0.304 INFO:Trainer:2024-13-05 15:47:18:hits@1_type: 0.215 INFO:Trainer:2024-13-05 15:47:18:hits@3_type: 0.33 INFO:Trainer:2024-13-05 15:47:18:hits@10_type: 0.485 INFO:EarlyStopping:2024-13-05 15:47:18:Validation score improved (0.431000 --> 0.432000). Saving model ... INFO:Trainer:2024-13-05 15:47:18:[cuda:0] Epoch [1100/10000] | Batchsize: 2048 | loss: 0.211951 | 2.67075 seconds/epoch DEBUG:connectionpool:2024-13-05 15:47:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:47:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:47:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:47:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:47:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:47:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:47:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:47:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:48:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:48:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:48:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:48:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:48:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:48:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:48:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:48:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:48:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:48:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:48:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:49:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:49:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:49:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:49:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:49:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:49:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:49:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:49:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:49:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:49:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:49:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:50:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:50:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:50:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:50:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:50:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:50:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:50:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:50:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:50:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:50:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:50:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:51:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:51:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:51:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:51:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:51:17:[cuda:0] Epoch 1200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 15:51:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:51:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:51:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:51:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:51:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:51:47:mr: 324.289 INFO:Trainer:2024-13-05 15:51:47:mrr: 0.269 INFO:Trainer:2024-13-05 15:51:47:hits@1: 0.187 INFO:Trainer:2024-13-05 15:51:47:hits@3: 0.297 INFO:Trainer:2024-13-05 15:51:47:hits@10: 0.432 INFO:Trainer:2024-13-05 15:51:47:mr_type: 120.62 INFO:Trainer:2024-13-05 15:51:47:mrr_type: 0.306 INFO:Trainer:2024-13-05 15:51:47:hits@1_type: 0.218 INFO:Trainer:2024-13-05 15:51:47:hits@3_type: 0.335 INFO:Trainer:2024-13-05 15:51:47:hits@10_type: 0.484 INFO:EarlyStopping:2024-13-05 15:51:47:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-13-05 15:51:47:[cuda:0] Epoch [1200/10000] | Batchsize: 2048 | loss: 0.211287 | 2.67163 seconds/epoch DEBUG:connectionpool:2024-13-05 15:51:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:51:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:51:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:52:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:52:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:52:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:52:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:52:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:52:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:52:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:52:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:52:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:52:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:52:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:53:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:53:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:53:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:53:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:53:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:53:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:53:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:53:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:53:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:53:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:53:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:54:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:54:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:54:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:54:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:54:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:54:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:54:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:54:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:54:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:54:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:54:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:54:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:55:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:55:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:55:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:55:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:55:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:55:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:55:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:55:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:55:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:55:52:[cuda:0] Epoch 1300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 15:55:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:55:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:56:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:56:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:56:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:56:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 15:56:24:mr: 316.881 INFO:Trainer:2024-13-05 15:56:24:mrr: 0.268 INFO:Trainer:2024-13-05 15:56:24:hits@1: 0.185 INFO:Trainer:2024-13-05 15:56:24:hits@3: 0.294 INFO:Trainer:2024-13-05 15:56:24:hits@10: 0.434 INFO:Trainer:2024-13-05 15:56:24:mr_type: 119.675 INFO:Trainer:2024-13-05 15:56:24:mrr_type: 0.307 INFO:Trainer:2024-13-05 15:56:24:hits@1_type: 0.218 INFO:Trainer:2024-13-05 15:56:24:hits@3_type: 0.334 INFO:Trainer:2024-13-05 15:56:24:hits@10_type: 0.485 INFO:EarlyStopping:2024-13-05 15:56:24:Validation score improved (0.432000 --> 0.434000). Saving model ... INFO:Trainer:2024-13-05 15:56:24:[cuda:0] Epoch [1300/10000] | Batchsize: 2048 | loss: 0.180422 | 2.67727 seconds/epoch DEBUG:connectionpool:2024-13-05 15:56:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:56:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:56:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:56:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:56:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:56:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:56:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:57:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:57:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:57:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:57:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:57:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:57:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:57:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:57:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:57:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:57:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:57:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:58:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:58:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:58:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:58:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:58:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:58:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:58:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:58:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:58:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:58:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:58:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:59:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:59:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:59:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:59:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:59:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:59:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:59:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:59:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:59:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:59:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:59:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 15:59:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:00:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:00:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:00:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:00:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:00:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:00:27:[cuda:0] Epoch 1400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 16:00:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:00:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:00:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:00:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:00:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:00:57:mr: 320.158 INFO:Trainer:2024-13-05 16:00:57:mrr: 0.269 INFO:Trainer:2024-13-05 16:00:57:hits@1: 0.186 INFO:Trainer:2024-13-05 16:00:57:hits@3: 0.295 INFO:Trainer:2024-13-05 16:00:57:hits@10: 0.435 INFO:Trainer:2024-13-05 16:00:57:mr_type: 119.465 INFO:Trainer:2024-13-05 16:00:57:mrr_type: 0.309 INFO:Trainer:2024-13-05 16:00:57:hits@1_type: 0.219 INFO:Trainer:2024-13-05 16:00:57:hits@3_type: 0.336 INFO:Trainer:2024-13-05 16:00:57:hits@10_type: 0.488 INFO:EarlyStopping:2024-13-05 16:00:57:Validation score improved (0.434000 --> 0.435000). Saving model ... INFO:Trainer:2024-13-05 16:00:57:[cuda:0] Epoch [1400/10000] | Batchsize: 2048 | loss: 0.198204 | 2.68293 seconds/epoch DEBUG:connectionpool:2024-13-05 16:00:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:01:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:01:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:01:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:01:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:01:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:01:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:01:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:01:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:01:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:01:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:01:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:02:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:02:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:02:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:02:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:02:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:02:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:02:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:02:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:02:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:02:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:02:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:03:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:03:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:03:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:03:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:03:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:03:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:03:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:03:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:03:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:03:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:03:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:03:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:04:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:04:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:04:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:04:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:04:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:04:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:04:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:04:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:04:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:04:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:04:55:[cuda:0] Epoch 1500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 16:04:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:05:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:05:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:05:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:05:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:05:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:05:24:mr: 319.915 INFO:Trainer:2024-13-05 16:05:24:mrr: 0.271 INFO:Trainer:2024-13-05 16:05:24:hits@1: 0.188 INFO:Trainer:2024-13-05 16:05:24:hits@3: 0.297 INFO:Trainer:2024-13-05 16:05:24:hits@10: 0.44 INFO:Trainer:2024-13-05 16:05:24:mr_type: 117.922 INFO:Trainer:2024-13-05 16:05:24:mrr_type: 0.311 INFO:Trainer:2024-13-05 16:05:24:hits@1_type: 0.22 INFO:Trainer:2024-13-05 16:05:24:hits@3_type: 0.338 INFO:Trainer:2024-13-05 16:05:24:hits@10_type: 0.493 INFO:EarlyStopping:2024-13-05 16:05:24:Validation score improved (0.435000 --> 0.440000). Saving model ... INFO:Trainer:2024-13-05 16:05:24:[cuda:0] Epoch [1500/10000] | Batchsize: 2048 | loss: 0.183093 | 2.68275 seconds/epoch DEBUG:connectionpool:2024-13-05 16:05:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:05:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:05:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:05:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:05:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:05:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:06:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:06:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:06:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:06:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:06:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:06:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:06:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:06:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:06:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:06:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:06:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:07:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:07:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:07:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:07:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:07:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:07:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:07:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:07:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:07:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:07:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:07:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:08:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:08:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:08:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:08:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:08:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:08:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:08:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:08:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:08:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:08:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:08:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:08:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:09:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:09:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:09:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:09:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:09:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:09:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:09:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:09:40:[cuda:0] Epoch 1600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 16:09:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:09:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:09:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:10:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:10:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:10:09:mr: 312.451 INFO:Trainer:2024-13-05 16:10:09:mrr: 0.273 INFO:Trainer:2024-13-05 16:10:09:hits@1: 0.189 INFO:Trainer:2024-13-05 16:10:09:hits@3: 0.299 INFO:Trainer:2024-13-05 16:10:09:hits@10: 0.438 INFO:Trainer:2024-13-05 16:10:09:mr_type: 116.78 INFO:Trainer:2024-13-05 16:10:09:mrr_type: 0.311 INFO:Trainer:2024-13-05 16:10:09:hits@1_type: 0.221 INFO:Trainer:2024-13-05 16:10:09:hits@3_type: 0.339 INFO:Trainer:2024-13-05 16:10:09:hits@10_type: 0.49 INFO:EarlyStopping:2024-13-05 16:10:09:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-13-05 16:10:09:[cuda:0] Epoch [1600/10000] | Batchsize: 2048 | loss: 0.199173 | 2.69322 seconds/epoch DEBUG:connectionpool:2024-13-05 16:10:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:10:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:10:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:10:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:10:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:10:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:10:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:10:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:10:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:10:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:11:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:11:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:11:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:11:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:11:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:11:31:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-13-05 16:11:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:11:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:11:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:11:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:11:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:11:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:12:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:12:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:12:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:12:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:12:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:12:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:12:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:12:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:12:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:12:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:12:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:13:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:13:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:13:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:13:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:13:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:13:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:13:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:13:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:13:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:13:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:13:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:14:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:14:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:14:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:14:14:[cuda:0] Epoch 1700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 16:14:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:14:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:14:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:14:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:14:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:14:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:14:42:mr: 312.065 INFO:Trainer:2024-13-05 16:14:42:mrr: 0.27 INFO:Trainer:2024-13-05 16:14:42:hits@1: 0.187 INFO:Trainer:2024-13-05 16:14:42:hits@3: 0.297 INFO:Trainer:2024-13-05 16:14:42:hits@10: 0.437 INFO:Trainer:2024-13-05 16:14:42:mr_type: 115.863 INFO:Trainer:2024-13-05 16:14:42:mrr_type: 0.309 INFO:Trainer:2024-13-05 16:14:42:hits@1_type: 0.219 INFO:Trainer:2024-13-05 16:14:42:hits@3_type: 0.336 INFO:Trainer:2024-13-05 16:14:42:hits@10_type: 0.491 INFO:EarlyStopping:2024-13-05 16:14:42:EarlyStopping counter: 2 / 2 INFO:Trainer:2024-13-05 16:14:42:[cuda:0] Send an early stopping signal INFO:Trainer:2024-13-05 16:14:42:[cuda:0] The model training is completed, taking a total of 4582.99834 seconds. INFO:Trainer:2024-13-05 16:14:43:[cuda:0] Model saved at ./model.pth. INFO:Trainer:2024-13-05 16:14:43:[cuda:0] The model starts evaluating in the test set. DEBUG:connectionpool:2024-13-05 16:14:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:14:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:14:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:15:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:15:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:15:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:15:16:mr: 327.896 INFO:Trainer:2024-13-05 16:15:16:mrr: 0.267 INFO:Trainer:2024-13-05 16:15:16:hits@1: 0.183 INFO:Trainer:2024-13-05 16:15:16:hits@3: 0.295 INFO:Trainer:2024-13-05 16:15:16:hits@10: 0.433 INFO:Trainer:2024-13-05 16:15:16:mr_type: 120.996 INFO:Trainer:2024-13-05 16:15:16:mrr_type: 0.305 INFO:Trainer:2024-13-05 16:15:16:hits@1_type: 0.215 INFO:Trainer:2024-13-05 16:15:16:hits@3_type: 0.334 INFO:Trainer:2024-13-05 16:15:16:hits@10_type: 0.486 wandb: - 0.014 MB of 0.014 MB uploaded DEBUG:connectionpool:2024-13-05 16:15:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: \ 0.014 MB of 0.121 MB uploaded wandb: | 0.025 MB of 0.121 MB uploaded wandb: / 0.121 MB of 0.121 MB uploaded wandb: - 0.121 MB of 0.121 MB uploaded wandb: \ 0.121 MB of 0.121 MB uploaded wandb: wandb: wandb: Run history: wandb: duration โ– wandb: test/hits@1 โ– wandb: test/hits@10 โ– wandb: test/hits@10_type โ– wandb: test/hits@1_type โ– wandb: test/hits@3 โ– wandb: test/hits@3_type โ– wandb: test/mr โ– wandb: test/mr_type โ– wandb: test/mrr โ– wandb: test/mrr_type โ– wandb: train/epoch โ–โ–โ–‚โ–‚โ–ƒโ–ƒโ–„โ–„โ–…โ–…โ–†โ–†โ–‡โ–‡โ–ˆโ–ˆ wandb: train/train_loss โ–ˆโ–„โ–ƒโ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ– wandb: val/epoch โ–โ–โ–‚โ–‚โ–ƒโ–ƒโ–„โ–„โ–…โ–…โ–…โ–†โ–†โ–‡โ–‡โ–ˆโ–ˆ wandb: val/hits@1 โ–โ–„โ–…โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@10 โ–โ–„โ–…โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@10_type โ–โ–„โ–…โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@1_type โ–โ–ƒโ–…โ–…โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@3 โ–โ–„โ–…โ–†โ–†โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@3_type โ–โ–„โ–…โ–†โ–†โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/mr โ–ˆโ–ƒโ–ƒโ–‚โ–โ–โ–โ–โ–โ–โ–‚โ–‚โ–โ–‚โ–‚โ–โ– wandb: val/mr_type โ–ˆโ–…โ–„โ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–โ–‚โ–โ–โ–โ–โ– wandb: val/mrr โ–โ–„โ–…โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–ˆโ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/mrr_type โ–โ–„โ–…โ–…โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: wandb: Run summary: wandb: duration 4582.99834 wandb: test/hits@1 0.183 wandb: test/hits@10 0.433 wandb: test/hits@10_type 0.486 wandb: test/hits@1_type 0.215 wandb: test/hits@3 0.295 wandb: test/hits@3_type 0.334 wandb: test/mr 327.896 wandb: test/mr_type 120.996 wandb: test/mrr 0.267 wandb: test/mrr_type 0.305 wandb: train/epoch 1600 wandb: train/train_loss 0.19917 wandb: val/epoch 1699 wandb: val/hits@1 0.187 wandb: val/hits@10 0.437 wandb: val/hits@10_type 0.491 wandb: val/hits@1_type 0.219 wandb: val/hits@3 0.297 wandb: val/hits@3_type 0.336 wandb: val/mr 312.065 wandb: val/mr_type 115.863 wandb: val/mrr 0.27 wandb: val/mrr_type 0.309 wandb: wandb: ๐Ÿš€ View run stilted-sweep-2 at: https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/yjp30i5f wandb: โญ๏ธ View project at: https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) wandb: Find logs at: ./wandb/run-20240513_145742-yjp30i5f/logs DEBUG:connectionpool:2024-13-05 16:15:23:Resetting dropped connection: o151352.ingest.sentry.io DEBUG:connectionpool:2024-13-05 16:15:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:15:24:https://o151352.ingest.sentry.io:443 "POST /api/4504800232407040/envelope/ HTTP/1.1" 200 0 DEBUG:connectionpool:2024-13-05 16:15:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: Agent Starting Run: 8wmd1asl with config: wandb: adv_temperature: 6 wandb: batch_size: 4096 wandb: dataloader: KGEDataLoader wandb: delta: 0.0001 wandb: device: cuda:0 wandb: dim: 100 wandb: ent_file: entity2id.txt wandb: epochs: 10000 wandb: in_path: ../../benchmarks/FB15K237/ wandb: l3_regul_rate: 0 wandb: log_interval: 100 wandb: loss: MarginLoss wandb: lr: 0.4069840319980425 wandb: lr_e: 0.5938195210091851 wandb: margin: 3 wandb: margin_e: 3 wandb: metric: hits@10 wandb: model: TransR wandb: neg_ent: 16 wandb: norm_flag: True wandb: num_workers: 16 wandb: opt_method: sgd wandb: opt_method_e: sgd wandb: p_norm: 1 wandb: patience: 2 wandb: prediction: all wandb: rand_init: False wandb: regul_rate: 0 wandb: rel_file: relation2id.txt wandb: save_path: ./model.pth wandb: strategy: NegativeSampling wandb: test_batch_size: 5 wandb: test_file: test2id.txt wandb: test_sampler: TradTestSampler wandb: tester: Tester wandb: train_file: train2id.txt wandb: train_sampler: BernSampler wandb: trainer: Trainer wandb: type_constrain: True wandb: use_early_stopping: True wandb: use_gpu: True wandb: use_tqdm: False wandb: valid_file: valid2id.txt wandb: valid_interval: 100 DEBUG:cmd:2024-13-05 16:15:30:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:cmd:2024-13-05 16:15:30:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-13-05 16:15:31:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-13-05 16:15:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:15:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:cmd:2024-13-05 16:15:31:Popen(['git', 'cat-file', '--batch-check'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=, shell=False, universal_newlines=False) wandb: - Waiting for wandb.init()... wandb: \ Waiting for wandb.init()... DEBUG:connectionpool:2024-13-05 16:15:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: Tracking run with wandb version 0.16.6 wandb: Run data is saved locally in /home/luyanfeng/my_code/github/pybind11-OpenKE/examples/TransR/wandb/run-20240513_161531-8wmd1asl wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run wild-sweep-3 wandb: โญ๏ธ View project at https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: ๐Ÿงน View sweep at https://wandb.ai/3555028709/pybind11-ke-sweeps/sweeps/bgb8gj47 wandb: ๐Ÿš€ View run at https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/8wmd1asl INFO:Trainer:2024-13-05 16:15:38:[cuda:0] Initialization completed, start model training. DEBUG:connectionpool:2024-13-05 16:15:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:15:41:[cuda:0] The model training is completed, taking a total of 3.49494 seconds. INFO:Trainer:2024-13-05 16:15:42:[cuda:0] Initialization completed, start model training. DEBUG:connectionpool:2024-13-05 16:15:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:15:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:15:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:16:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:16:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:16:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:16:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:16:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:16:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:16:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:16:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:16:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:16:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:16:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:16:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:17:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:17:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:17:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:17:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:17:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:17:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:17:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:17:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:17:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:17:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:18:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:18:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:18:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:18:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:18:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:18:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:18:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:18:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:18:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:18:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:18:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:19:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:19:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:19:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:19:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:19:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:19:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:19:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:19:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:19:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:19:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:19:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:20:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:20:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:20:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:20:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:20:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:20:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:20:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:20:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:20:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:20:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:20:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:20:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:21:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:21:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:21:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:21:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:21:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:21:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:21:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:21:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:21:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:21:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:21:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:22:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:22:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:22:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:22:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:22:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:22:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:22:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:22:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:22:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:22:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:22:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:22:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:23:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:23:09:[cuda:0] Epoch 100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 16:23:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:23:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:23:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:23:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:23:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:23:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:23:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:23:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:23:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:23:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:24:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:24:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:24:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:24:16:mr: 298.038 INFO:Trainer:2024-13-05 16:24:16:mrr: 0.311 INFO:Trainer:2024-13-05 16:24:16:hits@1: 0.22 INFO:Trainer:2024-13-05 16:24:16:hits@3: 0.344 INFO:Trainer:2024-13-05 16:24:16:hits@10: 0.492 INFO:Trainer:2024-13-05 16:24:16:mr_type: 111.991 INFO:Trainer:2024-13-05 16:24:16:mrr_type: 0.346 INFO:Trainer:2024-13-05 16:24:16:hits@1_type: 0.251 INFO:Trainer:2024-13-05 16:24:16:hits@3_type: 0.379 INFO:Trainer:2024-13-05 16:24:16:hits@10_type: 0.534 INFO:EarlyStopping:2024-13-05 16:24:16:Validation score improved (-inf --> 0.492000). Saving model ... INFO:Trainer:2024-13-05 16:24:16:[cuda:0] Epoch [ 100/10000] | Batchsize: 4096 | loss: 4.616153 | 4.46466 seconds/epoch DEBUG:connectionpool:2024-13-05 16:24:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:24:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:24:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:24:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:24:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:24:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:24:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:24:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:25:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:25:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:25:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:25:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:25:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:25:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:25:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:25:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:25:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:25:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:25:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:25:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:26:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:26:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:26:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:26:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:26:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:26:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:26:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:26:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:26:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:26:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:26:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:27:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:27:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:27:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:27:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:27:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:27:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:27:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:27:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:27:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:27:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:27:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:27:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:28:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:28:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:28:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:28:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:28:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:28:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:28:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:28:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:28:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:28:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:28:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:29:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:29:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:29:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:29:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:29:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:29:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:29:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:29:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:29:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:29:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:29:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:29:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:30:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:30:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:30:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:30:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:30:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:30:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:30:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:30:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:30:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:30:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:30:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:31:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:31:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:31:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:31:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:31:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:31:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:31:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:31:33:[cuda:0] Epoch 200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 16:31:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:31:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:31:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:31:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:31:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:32:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:32:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:32:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:32:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:32:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:32:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:32:31:mr: 332.978 INFO:Trainer:2024-13-05 16:32:31:mrr: 0.311 INFO:Trainer:2024-13-05 16:32:31:hits@1: 0.221 INFO:Trainer:2024-13-05 16:32:31:hits@3: 0.345 INFO:Trainer:2024-13-05 16:32:31:hits@10: 0.493 INFO:Trainer:2024-13-05 16:32:31:mr_type: 111.015 INFO:Trainer:2024-13-05 16:32:31:mrr_type: 0.348 INFO:Trainer:2024-13-05 16:32:31:hits@1_type: 0.254 INFO:Trainer:2024-13-05 16:32:31:hits@3_type: 0.382 INFO:Trainer:2024-13-05 16:32:31:hits@10_type: 0.537 INFO:EarlyStopping:2024-13-05 16:32:31:Validation score improved (0.492000 --> 0.493000). Saving model ... INFO:Trainer:2024-13-05 16:32:31:[cuda:0] Epoch [ 200/10000] | Batchsize: 4096 | loss: 3.476869 | 4.75399 seconds/epoch DEBUG:connectionpool:2024-13-05 16:32:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:32:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:32:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:32:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:32:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:33:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:33:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:33:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:33:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:33:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:33:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:33:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:33:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:33:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:33:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:33:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:33:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:34:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:34:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:34:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:34:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:34:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:34:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:34:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:34:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:34:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:34:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:34:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:35:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:35:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:35:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:35:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:35:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:35:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:35:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:35:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:35:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:35:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:35:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:35:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:36:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:36:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:36:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:36:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:36:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:36:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:36:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:36:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:36:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:36:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:36:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:37:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:37:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:37:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:37:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:37:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:37:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:37:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:37:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:37:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:37:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:37:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:37:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:38:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:38:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:38:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:38:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:38:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:38:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:38:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:38:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:38:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:38:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:38:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:39:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:39:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:39:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:39:17:[cuda:0] Epoch 300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 16:39:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:39:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:39:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:39:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:39:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:39:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:39:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:39:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:39:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:40:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:40:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:40:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:40:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:40:25:mr: 359.74 INFO:Trainer:2024-13-05 16:40:25:mrr: 0.308 INFO:Trainer:2024-13-05 16:40:25:hits@1: 0.218 INFO:Trainer:2024-13-05 16:40:25:hits@3: 0.341 INFO:Trainer:2024-13-05 16:40:25:hits@10: 0.486 INFO:Trainer:2024-13-05 16:40:25:mr_type: 112.211 INFO:Trainer:2024-13-05 16:40:25:mrr_type: 0.345 INFO:Trainer:2024-13-05 16:40:25:hits@1_type: 0.251 INFO:Trainer:2024-13-05 16:40:25:hits@3_type: 0.38 INFO:Trainer:2024-13-05 16:40:25:hits@10_type: 0.532 INFO:EarlyStopping:2024-13-05 16:40:25:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-13-05 16:40:25:[cuda:0] Epoch [ 300/10000] | Batchsize: 4096 | loss: 2.876500 | 4.71701 seconds/epoch DEBUG:connectionpool:2024-13-05 16:40:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:40:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:40:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:40:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:40:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:40:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:40:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:41:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:41:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:41:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:41:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:41:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:41:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:41:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:41:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:41:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:41:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:41:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:41:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:42:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:42:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:42:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:42:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:42:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:42:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:42:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:42:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:42:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:42:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:42:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:43:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:43:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:43:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:43:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:43:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:43:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:43:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:43:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:43:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:43:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:43:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:44:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:44:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:44:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:44:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:44:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:44:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:44:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:44:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:44:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:44:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:44:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:45:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:45:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:45:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:45:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:45:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:45:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:45:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:45:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:45:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:45:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:45:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:45:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:46:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:46:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:46:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:46:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:46:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:46:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:46:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:46:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:46:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:46:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:46:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:47:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:47:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:47:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:47:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:47:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:47:26:[cuda:0] Epoch 400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-13-05 16:47:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:47:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:47:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:47:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:47:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:47:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:47:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:48:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:48:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:48:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:48:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:48:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:48:27:mr: 381.375 INFO:Trainer:2024-13-05 16:48:27:mrr: 0.305 INFO:Trainer:2024-13-05 16:48:27:hits@1: 0.216 INFO:Trainer:2024-13-05 16:48:27:hits@3: 0.339 INFO:Trainer:2024-13-05 16:48:27:hits@10: 0.482 INFO:Trainer:2024-13-05 16:48:27:mr_type: 114.392 INFO:Trainer:2024-13-05 16:48:27:mrr_type: 0.344 INFO:Trainer:2024-13-05 16:48:27:hits@1_type: 0.251 INFO:Trainer:2024-13-05 16:48:27:hits@3_type: 0.377 INFO:Trainer:2024-13-05 16:48:27:hits@10_type: 0.53 INFO:EarlyStopping:2024-13-05 16:48:27:EarlyStopping counter: 2 / 2 INFO:Trainer:2024-13-05 16:48:27:[cuda:0] Send an early stopping signal INFO:Trainer:2024-13-05 16:48:27:[cuda:0] The model training is completed, taking a total of 1903.45028 seconds. INFO:Trainer:2024-13-05 16:48:27:[cuda:0] Model saved at ./model.pth. INFO:Trainer:2024-13-05 16:48:27:[cuda:0] The model starts evaluating in the test set. DEBUG:connectionpool:2024-13-05 16:48:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:48:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:48:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:48:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:48:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:48:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:49:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:49:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:49:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:49:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:49:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:49:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:49:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-13-05 16:49:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-13-05 16:49:39:mr: 395.375 INFO:Trainer:2024-13-05 16:49:39:mrr: 0.303 INFO:Trainer:2024-13-05 16:49:39:hits@1: 0.213 INFO:Trainer:2024-13-05 16:49:39:hits@3: 0.338 INFO:Trainer:2024-13-05 16:49:39:hits@10: 0.48 INFO:Trainer:2024-13-05 16:49:39:mr_type: 118.118 INFO:Trainer:2024-13-05 16:49:39:mrr_type: 0.341 INFO:Trainer:2024-13-05 16:49:39:hits@1_type: 0.247 INFO:Trainer:2024-13-05 16:49:39:hits@3_type: 0.376 INFO:Trainer:2024-13-05 16:49:39:hits@10_type: 0.526 wandb: - 0.014 MB of 0.014 MB uploaded wandb: \ 0.014 MB of 0.014 MB uploaded wandb: | 0.055 MB of 0.066 MB uploaded DEBUG:connectionpool:2024-13-05 16:49:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: / 0.059 MB of 0.066 MB uploaded wandb: - 0.059 MB of 0.066 MB uploaded wandb: \ 0.059 MB of 0.066 MB uploaded wandb: | 0.066 MB of 0.066 MB uploaded wandb: wandb: wandb: Run history: wandb: duration โ– wandb: test/hits@1 โ– wandb: test/hits@10 โ– wandb: test/hits@10_type โ– wandb: test/hits@1_type โ– wandb: test/hits@3 โ– wandb: test/hits@3_type โ– wandb: test/mr โ– wandb: test/mr_type โ– wandb: test/mrr โ– wandb: test/mrr_type โ– wandb: train/epoch โ–โ–…โ–ˆ wandb: train/train_loss โ–ˆโ–ƒโ– wandb: val/epoch โ–โ–ƒโ–†โ–ˆ wandb: val/hits@1 โ–‡โ–ˆโ–„โ– wandb: val/hits@10 โ–‡โ–ˆโ–„โ– wandb: val/hits@10_type โ–…โ–ˆโ–ƒโ– wandb: val/hits@1_type โ–โ–ˆโ–โ– wandb: val/hits@3 โ–‡โ–ˆโ–ƒโ– wandb: val/hits@3_type โ–„โ–ˆโ–…โ– wandb: val/mr โ–โ–„โ–†โ–ˆ wandb: val/mr_type โ–ƒโ–โ–ƒโ–ˆ wandb: val/mrr โ–ˆโ–ˆโ–…โ– wandb: val/mrr_type โ–„โ–ˆโ–ƒโ– wandb: wandb: Run summary: wandb: duration 1903.45028 wandb: test/hits@1 0.213 wandb: test/hits@10 0.48 wandb: test/hits@10_type 0.526 wandb: test/hits@1_type 0.247 wandb: test/hits@3 0.338 wandb: test/hits@3_type 0.376 wandb: test/mr 395.375 wandb: test/mr_type 118.118 wandb: test/mrr 0.303 wandb: test/mrr_type 0.341 wandb: train/epoch 300 wandb: train/train_loss 2.8765 wandb: val/epoch 399 wandb: val/hits@1 0.216 wandb: val/hits@10 0.482 wandb: val/hits@10_type 0.53 wandb: val/hits@1_type 0.251 wandb: val/hits@3 0.339 wandb: val/hits@3_type 0.377 wandb: val/mr 381.375 wandb: val/mr_type 114.392 wandb: val/mrr 0.305 wandb: val/mrr_type 0.344 wandb: wandb: ๐Ÿš€ View run wild-sweep-3 at: https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/8wmd1asl wandb: โญ๏ธ View project at: https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) wandb: Find logs at: ./wandb/run-20240513_161531-8wmd1asl/logs DEBUG:connectionpool:2024-13-05 16:49:46:Resetting dropped connection: o151352.ingest.sentry.io DEBUG:connectionpool:2024-13-05 16:49:47:https://o151352.ingest.sentry.io:443 "POST /api/4504800232407040/envelope/ HTTP/1.1" 200 0