DEBUG:cmd:2024-05-23 13:55:12:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:cmd:2024-05-23 13:55:12:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-05-23 13:55:13:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-05-23 13:55:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:55:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: Currently logged in as: 3555028709. Use `wandb login --relogin` to force relogin DEBUG:connectionpool:2024-05-23 13:55:14:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-05-23 13:55:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:pyagent:2024-05-23 13:55:14:Starting sweep agent: entity=None, project=None, count=3 DEBUG:pyagent:2024-05-23 13:55:14:Agent._setup() DEBUG:pyagent:2024-05-23 13:55:14:Agent._register() DEBUG:connectionpool:2024-05-23 13:55:14:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-05-23 13:55:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:pyagent:2024-05-23 13:55:15:agent_id = QWdlbnQ6bHZydnhyNXo= DEBUG:connectionpool:2024-05-23 13:55:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:pyagent:2024-05-23 13:55:15:Job received: Job(343tmdcx,{'batch_size': {'value': 4096}, 'dataloader': {'value': 'KGEDataLoader'}, 'decoder_model': {'value': 'ConvE'}, 'delta': {'value': 0.0001}, 'device': {'value': 'cuda:0'}, 'dim': {'value': 100}, 'ent_file': {'value': 'entity2id.txt'}, 'epochs': {'value': 10000}, 'fet_drop': {'value': 0.2}, 'hid_drop': {'value': 0.3}, 'in_path': {'value': '../../benchmarks/FB15K237/'}, 'log_interval': {'value': 100}, 'loss': {'value': 'CompGCNLoss'}, 'lr': {'value': 0.0041602769746015445}, 'margin': {'value': 40}, 'metric': {'value': 'hits@10'}, 'model': {'value': 'CompGCN'}, 'neg_ent': {'value': 1}, 'num_workers': {'value': 16}, 'opn': {'value': 'mult'}, 'opt_method': {'value': 'sgd'}, 'patience': {'value': 2}, 'prediction': {'value': 'tail'}, 'rel_file': {'value': 'relation2id.txt'}, 'save_path': {'value': './model.pth'}, 'smoothing': {'value': 0.1}, 'strategy': {'value': 'CompGCNSampling'}, 'test_batch_size': {'value': 256}, 'test_file': {'value': 'test2id.txt'}, 'test_sampler': {'value': 'CompGCNTestSampler'}, 'tester': {'value': 'Tester'}, 'train_file': {'value': 'train2id.txt'}, 'train_sampler': {'value': 'CompGCNSampler'}, 'trainer': {'value': 'Trainer'}, 'type_constrain': {'value': True}, 'use_early_stopping': {'value': True}, 'use_gpu': {'value': True}, 'use_tqdm': {'value': False}, 'valid_file': {'value': 'valid2id.txt'}, 'valid_interval': {'value': 100}}) DEBUG:pyagent:2024-05-23 13:55:15:Spawning new thread for run 343tmdcx. wandb: Agent Starting Run: 343tmdcx with config: wandb: batch_size: 4096 wandb: dataloader: KGEDataLoader wandb: decoder_model: ConvE wandb: delta: 0.0001 wandb: device: cuda:0 wandb: dim: 100 wandb: ent_file: entity2id.txt wandb: epochs: 10000 wandb: fet_drop: 0.2 wandb: hid_drop: 0.3 wandb: in_path: ../../benchmarks/FB15K237/ wandb: log_interval: 100 wandb: loss: CompGCNLoss wandb: lr: 0.0041602769746015445 wandb: margin: 40 wandb: metric: hits@10 wandb: model: CompGCN wandb: neg_ent: 1 wandb: num_workers: 16 wandb: opn: mult wandb: opt_method: sgd wandb: patience: 2 wandb: prediction: tail wandb: rel_file: relation2id.txt wandb: save_path: ./model.pth wandb: smoothing: 0.1 wandb: strategy: CompGCNSampling wandb: test_batch_size: 256 wandb: test_file: test2id.txt wandb: test_sampler: CompGCNTestSampler wandb: tester: Tester wandb: train_file: train2id.txt wandb: train_sampler: CompGCNSampler wandb: trainer: Trainer wandb: type_constrain: True wandb: use_early_stopping: True wandb: use_gpu: True wandb: use_tqdm: False wandb: valid_file: valid2id.txt wandb: valid_interval: 100 DEBUG:cmd:2024-05-23 13:55:16:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:cmd:2024-05-23 13:55:16:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-05-23 13:55:16:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-05-23 13:55:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:55:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:cmd:2024-05-23 13:55:17:Popen(['git', 'cat-file', '--batch-check'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-05-23 13:55:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: Tracking run with wandb version 0.16.6 wandb: Run data is saved locally in /home/luyanfeng/my_code/github/pybind11-OpenKE/examples/CompGCN/wandb/run-20240523_135517-343tmdcx wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run confused-sweep-1 wandb: โญ๏ธ View project at https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: ๐Ÿงน View sweep at https://wandb.ai/3555028709/pybind11-ke-sweeps/sweeps/fnx5v46t wandb: ๐Ÿš€ View run at https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/343tmdcx DEBUG:connectionpool:2024-05-23 13:55:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 13:55:30:[cuda:0] Initialization completed, start model training. data_loader_config: {'batch_size': {'values': [512, 1024, 2048, 4096]}, 'dataloader': {'value': 'KGEDataLoader'}, 'ent_file': {'value': 'entity2id.txt'}, 'in_path': {'value': './'}, 'neg_ent': {'values': [1, 4, 16, 64]}, 'num_workers': {'value': 16}, 'rel_file': {'value': 'relation2id.txt'}, 'test_batch_size': {'value': 30}, 'test_file': {'value': 'test2id.txt'}, 'test_sampler': {'value': 'TradTestSampler'}, 'train_file': {'value': 'train2id.txt'}, 'train_sampler': {'value': 'BernSampler'}, 'type_constrain': {'value': True}, 'valid_file': {'value': 'valid2id.txt'}} kge_config: {'decoder_model': {'value': 'ConvE'}, 'dim': {'values': [100, 150, 200]}, 'fet_drop': {'value': 0.2}, 'hid_drop': {'value': 0.3}, 'margin': {'value': 40.0}, 'model': {'value': 'CompGCN'}, 'opn': {'value': 'mult'}} loss_config: {'loss': {'value': 'CompGCNLoss'}} strategy_config: {'smoothing': {'value': 0.1}, 'strategy': {'value': 'CompGCNSampling'}} tester_config: {'device': {'value': 'cuda:0'}, 'prediction': {'value': 'all'}, 'tester': {'value': 'Tester'}, 'use_gpu': {'value': True}, 'use_tqdm': {'value': False}} trainer_config: {'delta': {'value': 0.0001}, 'epochs': {'value': 10000}, 'log_interval': {'value': 100}, 'lr': {'distribution': 'uniform', 'max': 1.0, 'min': 1e-05}, 'metric': {'value': 'hits@10'}, 'opt_method': {'values': ['adam', 'adagrad', 'sgd']}, 'patience': {'value': 2}, 'save_path': {'value': './model.pth'}, 'trainer': {'value': 'Trainer'}, 'use_early_stopping': {'value': True}, 'valid_interval': {'value': 100}} sweep_config: {'method': 'bayes', 'metric': {'goal': 'maximize', 'name': 'val/hits@10'}, 'name': 'CompGCN_FB15K237', 'parameters': {'batch_size': {'values': [512, 1024, 2048, 4096]}, 'dataloader': {'value': 'KGEDataLoader'}, 'decoder_model': {'value': 'ConvE'}, 'delta': {'value': 0.0001}, 'device': {'value': 'cuda:0'}, 'dim': {'values': [100, 150, 200]}, 'ent_file': {'value': 'entity2id.txt'}, 'epochs': {'value': 10000}, 'fet_drop': {'value': 0.2}, 'hid_drop': {'value': 0.3}, 'in_path': {'value': '../../benchmarks/FB15K237/'}, 'log_interval': {'value': 100}, 'loss': {'value': 'CompGCNLoss'}, 'lr': {'distribution': 'uniform', 'max': 0.01, 'min': 0}, 'margin': {'value': 40.0}, 'metric': {'value': 'hits@10'}, 'model': {'value': 'CompGCN'}, 'neg_ent': {'value': 1}, 'num_workers': {'value': 16}, 'opn': {'value': 'mult'}, 'opt_method': {'values': ['adam', 'adagrad', 'sgd']}, 'patience': {'value': 2}, 'prediction': {'value': 'tail'}, 'rel_file': {'value': 'relation2id.txt'}, 'save_path': {'value': './model.pth'}, 'smoothing': {'value': 0.1}, 'strategy': {'value': 'CompGCNSampling'}, 'test_batch_size': {'value': 256}, 'test_file': {'value': 'test2id.txt'}, 'test_sampler': {'value': 'CompGCNTestSampler'}, 'tester': {'value': 'Tester'}, 'train_file': {'value': 'train2id.txt'}, 'train_sampler': {'value': 'CompGCNSampler'}, 'trainer': {'value': 'Trainer'}, 'type_constrain': {'value': True}, 'use_early_stopping': {'value': True}, 'use_gpu': {'value': True}, 'use_tqdm': {'value': False}, 'valid_file': {'value': 'valid2id.txt'}, 'valid_interval': {'value': 100}}} Create sweep with ID: fnx5v46t Sweep URL: https://wandb.ai/3555028709/pybind11-ke-sweeps/sweeps/fnx5v46t DEBUG:connectionpool:2024-05-23 13:55:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:55:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:55:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:55:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:55:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:55:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:56:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:56:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:56:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:56:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:56:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:56:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:56:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:56:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:56:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:56:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:56:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:57:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:57:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:57:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:57:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:57:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:57:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:57:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:57:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:57:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:57:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:57:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:57:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:58:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:58:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:58:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:58:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:58:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:58:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:58:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:58:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:58:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:58:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:58:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:59:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:59:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:59:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:59:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:59:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:59:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:59:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:59:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:59:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:59:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 13:59:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:00:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:00:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:00:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:00:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:00:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:00:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:00:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:00:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:00:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:00:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:00:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:00:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:01:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:01:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:01:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:01:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:01:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:01:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:01:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:01:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:01:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:01:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:01:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:02:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:02:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:02:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:02:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:02:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:02:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:02:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:02:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:02:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:02:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:02:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:03:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:03:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:03:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:03:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:03:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:03:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:03:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:03:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:03:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:03:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:03:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:03:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:04:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:04:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:04:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:04:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:04:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:04:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:04:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:04:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:04:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:04:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:04:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:05:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:05:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:05:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:05:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:05:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:05:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:05:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 14:05:37:[cuda:0] Epoch 100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 14:05:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 14:05:44:mr: 6797.763 INFO:Trainer:2024-05-23 14:05:44:mrr: 0.001 INFO:Trainer:2024-05-23 14:05:44:hits@1: 0.0 INFO:Trainer:2024-05-23 14:05:44:hits@3: 0.001 INFO:Trainer:2024-05-23 14:05:44:hits@10: 0.002 INFO:Trainer:2024-05-23 14:05:44:mr_type: 375.367 INFO:Trainer:2024-05-23 14:05:44:mrr_type: 0.111 INFO:Trainer:2024-05-23 14:05:44:hits@1_type: 0.069 INFO:Trainer:2024-05-23 14:05:44:hits@3_type: 0.112 INFO:Trainer:2024-05-23 14:05:44:hits@10_type: 0.183 INFO:EarlyStopping:2024-05-23 14:05:44:Validation score improved (-inf --> 0.002000). Saving model ... INFO:Trainer:2024-05-23 14:05:44:[cuda:0] Epoch [ 100/10000] | Batchsize: 4096 | loss: 0.120855 | 6.07382 seconds/epoch DEBUG:connectionpool:2024-05-23 14:05:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:05:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:05:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:06:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:06:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:06:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:06:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:06:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:06:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:06:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:06:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:06:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:06:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:06:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:06:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:07:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:07:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:07:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:07:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:07:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:07:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:07:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:07:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:07:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:07:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:07:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:08:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:08:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:08:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:08:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:08:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:08:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:08:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:08:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:08:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:08:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:08:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:09:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:09:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:09:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:09:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:09:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:09:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:09:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:09:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:09:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:09:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:09:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:09:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:10:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:10:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:10:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:10:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:10:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:10:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:10:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:10:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:10:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:10:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:10:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:11:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:11:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:11:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:11:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:11:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:11:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:11:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:11:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:11:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:11:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:11:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:12:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:12:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:12:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:12:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:12:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:12:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:12:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:12:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:12:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:12:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:12:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:13:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:13:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:13:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:13:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:13:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:13:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:13:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:13:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:13:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:13:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:13:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:13:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:14:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:14:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:14:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:14:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:14:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:14:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:14:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:14:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:14:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:14:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:14:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:15:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:15:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:15:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:15:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:15:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:15:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:15:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:15:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:15:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:15:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:15:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 14:15:57:[cuda:0] Epoch 200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 14:16:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 14:16:03:mr: 6620.583 INFO:Trainer:2024-05-23 14:16:03:mrr: 0.008 INFO:Trainer:2024-05-23 14:16:03:hits@1: 0.005 INFO:Trainer:2024-05-23 14:16:03:hits@3: 0.008 INFO:Trainer:2024-05-23 14:16:03:hits@10: 0.013 INFO:Trainer:2024-05-23 14:16:03:mr_type: 374.347 INFO:Trainer:2024-05-23 14:16:03:mrr_type: 0.121 INFO:Trainer:2024-05-23 14:16:03:hits@1_type: 0.081 INFO:Trainer:2024-05-23 14:16:03:hits@3_type: 0.121 INFO:Trainer:2024-05-23 14:16:03:hits@10_type: 0.192 INFO:EarlyStopping:2024-05-23 14:16:03:Validation score improved (0.002000 --> 0.013000). Saving model ... INFO:Trainer:2024-05-23 14:16:03:[cuda:0] Epoch [ 200/10000] | Batchsize: 4096 | loss: 0.115806 | 6.13558 seconds/epoch DEBUG:connectionpool:2024-05-23 14:16:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:16:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:16:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:16:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:16:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:16:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:16:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:16:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:16:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:16:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:16:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:17:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:17:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:17:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:17:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:17:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:17:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:17:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:17:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:17:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:17:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:17:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:18:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:18:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:18:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:18:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:18:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:18:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:18:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:18:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:18:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:18:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:18:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:19:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:19:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:19:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:19:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:19:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:19:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:19:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:19:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:19:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:19:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:19:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:19:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:20:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:20:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:20:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:20:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:20:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:20:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:20:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:20:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:20:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:20:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:20:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:21:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:21:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:21:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:21:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:21:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:21:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:21:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:21:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:21:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:21:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:21:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:21:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:22:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:22:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:22:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:22:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:22:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:22:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:22:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:22:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:22:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:22:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:22:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:23:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:23:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:23:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:23:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:23:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:23:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:23:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:23:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:23:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:23:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:23:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:24:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:24:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:24:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:24:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:24:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:24:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:24:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:24:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:24:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:24:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:24:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:24:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:25:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:25:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:25:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:25:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:25:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:25:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:25:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:25:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:25:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:25:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:25:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:26:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:26:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:26:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:26:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:26:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 14:26:24:[cuda:0] Epoch 300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 14:26:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 14:26:30:mr: 6481.807 INFO:Trainer:2024-05-23 14:26:30:mrr: 0.021 INFO:Trainer:2024-05-23 14:26:30:hits@1: 0.016 INFO:Trainer:2024-05-23 14:26:30:hits@3: 0.021 INFO:Trainer:2024-05-23 14:26:30:hits@10: 0.028 INFO:Trainer:2024-05-23 14:26:30:mr_type: 373.456 INFO:Trainer:2024-05-23 14:26:30:mrr_type: 0.13 INFO:Trainer:2024-05-23 14:26:30:hits@1_type: 0.091 INFO:Trainer:2024-05-23 14:26:30:hits@3_type: 0.129 INFO:Trainer:2024-05-23 14:26:30:hits@10_type: 0.199 INFO:EarlyStopping:2024-05-23 14:26:30:Validation score improved (0.013000 --> 0.028000). Saving model ... INFO:Trainer:2024-05-23 14:26:31:[cuda:0] Epoch [ 300/10000] | Batchsize: 4096 | loss: 0.112285 | 6.18196 seconds/epoch DEBUG:connectionpool:2024-05-23 14:26:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:26:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:26:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:26:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:26:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:27:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:27:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:27:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:27:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:27:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:27:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:27:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:27:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:27:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:27:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:27:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:27:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:28:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:28:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:28:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:28:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:28:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:28:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:28:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:28:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:28:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:28:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:28:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:29:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:29:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:29:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:29:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:29:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:29:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:29:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:29:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:29:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:29:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:29:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:30:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:30:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:30:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:30:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:30:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:30:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:30:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:30:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:30:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:30:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:30:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:30:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:31:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:31:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:31:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:31:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:31:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:31:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:31:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:31:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:31:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:31:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:31:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:32:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:32:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:32:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:32:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:32:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:32:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:32:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:32:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:32:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:32:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:32:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:33:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:33:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:33:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:33:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:33:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:33:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:33:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:33:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:33:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:33:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:33:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:33:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:34:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:34:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:34:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:34:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:34:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:34:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:34:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:34:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:34:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:34:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:34:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:35:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:35:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:35:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:35:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:35:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:35:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:35:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:35:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:35:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:35:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:35:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:36:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:36:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:36:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:36:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:36:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:36:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:36:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:36:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:36:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 14:36:45:[cuda:0] Epoch 400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 14:36:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 14:36:51:mr: 6365.753 INFO:Trainer:2024-05-23 14:36:51:mrr: 0.033 INFO:Trainer:2024-05-23 14:36:51:hits@1: 0.028 INFO:Trainer:2024-05-23 14:36:51:hits@3: 0.034 INFO:Trainer:2024-05-23 14:36:51:hits@10: 0.041 INFO:Trainer:2024-05-23 14:36:51:mr_type: 372.597 INFO:Trainer:2024-05-23 14:36:51:mrr_type: 0.137 INFO:Trainer:2024-05-23 14:36:51:hits@1_type: 0.099 INFO:Trainer:2024-05-23 14:36:51:hits@3_type: 0.135 INFO:Trainer:2024-05-23 14:36:51:hits@10_type: 0.203 INFO:EarlyStopping:2024-05-23 14:36:51:Validation score improved (0.028000 --> 0.041000). Saving model ... INFO:Trainer:2024-05-23 14:36:51:[cuda:0] Epoch [ 400/10000] | Batchsize: 4096 | loss: 0.109153 | 6.18875 seconds/epoch DEBUG:connectionpool:2024-05-23 14:36:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:36:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:37:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:37:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:37:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:37:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:37:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:37:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:37:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:37:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:37:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:37:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:37:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:38:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:38:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:38:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:38:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:38:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:38:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:38:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:38:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:38:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:38:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:38:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:39:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:39:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:39:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:39:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:39:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:39:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:39:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:39:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:39:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:39:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:39:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:39:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:40:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:40:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:40:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:40:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:40:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:40:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:40:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:40:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:40:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:40:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:40:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:41:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:41:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:41:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:41:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:41:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:41:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:41:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:41:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:41:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:41:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:41:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:42:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:42:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:42:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:42:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:42:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:42:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:42:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:42:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:42:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:42:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:42:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:43:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:43:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:43:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:43:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:43:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:43:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:43:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:43:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:43:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:43:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:43:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:44:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:44:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:44:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:44:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:44:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:44:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:44:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:44:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:44:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:44:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:44:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:44:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:45:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:45:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:45:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:45:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:45:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:45:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:45:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:45:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:45:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:45:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:45:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:46:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:46:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:46:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:46:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:46:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:46:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:46:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:46:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:46:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:46:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:46:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:46:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:47:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 14:47:05:[cuda:0] Epoch 500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 14:47:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 14:47:11:mr: 6256.264 INFO:Trainer:2024-05-23 14:47:11:mrr: 0.045 INFO:Trainer:2024-05-23 14:47:11:hits@1: 0.04 INFO:Trainer:2024-05-23 14:47:11:hits@3: 0.046 INFO:Trainer:2024-05-23 14:47:11:hits@10: 0.052 INFO:Trainer:2024-05-23 14:47:11:mr_type: 371.864 INFO:Trainer:2024-05-23 14:47:11:mrr_type: 0.143 INFO:Trainer:2024-05-23 14:47:11:hits@1_type: 0.105 INFO:Trainer:2024-05-23 14:47:11:hits@3_type: 0.14 INFO:Trainer:2024-05-23 14:47:11:hits@10_type: 0.208 INFO:EarlyStopping:2024-05-23 14:47:11:Validation score improved (0.041000 --> 0.052000). Saving model ... INFO:Trainer:2024-05-23 14:47:11:[cuda:0] Epoch [ 500/10000] | Batchsize: 4096 | loss: 0.106846 | 6.19038 seconds/epoch DEBUG:connectionpool:2024-05-23 14:47:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:47:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:47:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:47:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:47:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:47:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:47:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:47:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:47:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:48:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:48:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:48:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:48:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:48:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:48:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:48:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:48:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:48:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:48:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:48:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:49:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:49:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:49:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:49:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:49:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:49:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:49:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:49:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:49:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:49:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:49:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:49:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:50:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:50:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:50:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:50:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:50:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:50:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:50:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:50:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:50:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:50:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:50:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:51:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:51:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:51:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:51:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:51:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:51:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:51:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:51:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:51:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:51:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:51:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:52:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:52:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:52:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:52:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:52:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:52:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:52:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:52:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:52:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:52:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:52:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:52:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:53:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:53:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:53:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:53:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:53:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:53:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:53:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:53:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:53:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:53:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:53:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:54:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:54:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:54:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:54:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:54:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:54:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:54:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:54:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:54:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:54:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:54:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:55:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:55:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:55:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:55:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:55:22:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-23 14:55:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:55:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:55:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:55:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:55:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:55:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:55:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:55:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:56:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:56:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:56:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:56:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:56:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:56:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:56:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:56:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:56:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:56:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:56:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:57:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:57:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:57:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:57:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:57:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 14:57:28:[cuda:0] Epoch 600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 14:57:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:57:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 14:57:35:mr: 6155.342 INFO:Trainer:2024-05-23 14:57:35:mrr: 0.052 INFO:Trainer:2024-05-23 14:57:35:hits@1: 0.048 INFO:Trainer:2024-05-23 14:57:35:hits@3: 0.053 INFO:Trainer:2024-05-23 14:57:35:hits@10: 0.058 INFO:Trainer:2024-05-23 14:57:35:mr_type: 371.153 INFO:Trainer:2024-05-23 14:57:35:mrr_type: 0.147 INFO:Trainer:2024-05-23 14:57:35:hits@1_type: 0.109 INFO:Trainer:2024-05-23 14:57:35:hits@3_type: 0.144 INFO:Trainer:2024-05-23 14:57:35:hits@10_type: 0.214 INFO:EarlyStopping:2024-05-23 14:57:35:Validation score improved (0.052000 --> 0.058000). Saving model ... INFO:Trainer:2024-05-23 14:57:35:[cuda:0] Epoch [ 600/10000] | Batchsize: 4096 | loss: 0.104606 | 6.19789 seconds/epoch DEBUG:connectionpool:2024-05-23 14:57:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:57:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:57:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:57:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:58:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:58:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:58:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:58:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:58:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:58:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:58:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:58:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:58:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:58:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:58:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:58:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:59:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:59:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:59:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:59:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:59:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:59:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:59:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:59:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:59:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:59:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 14:59:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:00:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:00:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:00:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:00:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:00:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:00:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:00:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:00:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:00:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:00:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:00:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:01:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:01:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:01:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:01:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:01:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:01:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:01:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:01:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:01:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:01:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:01:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:01:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:02:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:02:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:02:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:02:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:02:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:02:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:02:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:02:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:02:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:02:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:02:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:03:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:03:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:03:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:03:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:03:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:03:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:03:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:03:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:03:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:03:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:03:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:03:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:04:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:04:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:04:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:04:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:04:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:04:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:04:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:04:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:04:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:04:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:04:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:05:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:05:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:05:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:05:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:05:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:05:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:05:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:05:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:05:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:05:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:05:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:06:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:06:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:06:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:06:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:06:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:06:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:06:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:06:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:06:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:06:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:06:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:06:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:07:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:07:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:07:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:07:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:07:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:07:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:07:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:07:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:07:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 15:07:49:[cuda:0] Epoch 700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 15:07:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 15:07:56:mr: 6059.657 INFO:Trainer:2024-05-23 15:07:56:mrr: 0.057 INFO:Trainer:2024-05-23 15:07:56:hits@1: 0.053 INFO:Trainer:2024-05-23 15:07:56:hits@3: 0.057 INFO:Trainer:2024-05-23 15:07:56:hits@10: 0.062 INFO:Trainer:2024-05-23 15:07:56:mr_type: 370.434 INFO:Trainer:2024-05-23 15:07:56:mrr_type: 0.15 INFO:Trainer:2024-05-23 15:07:56:hits@1_type: 0.112 INFO:Trainer:2024-05-23 15:07:56:hits@3_type: 0.149 INFO:Trainer:2024-05-23 15:07:56:hits@10_type: 0.218 INFO:EarlyStopping:2024-05-23 15:07:56:Validation score improved (0.058000 --> 0.062000). Saving model ... DEBUG:connectionpool:2024-05-23 15:07:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 15:07:56:[cuda:0] Epoch [ 700/10000] | Batchsize: 4096 | loss: 0.102812 | 6.19984 seconds/epoch DEBUG:connectionpool:2024-05-23 15:08:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:08:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:08:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:08:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:08:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:08:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:08:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:08:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:08:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:08:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:08:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:08:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:09:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:09:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:09:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:09:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:09:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:09:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:09:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:09:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:09:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:09:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:09:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:10:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:10:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:10:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:10:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:10:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:10:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:10:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:10:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:10:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:10:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:10:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:11:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:11:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:11:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:11:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:11:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:11:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:11:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:11:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:11:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:11:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:11:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:11:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:12:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:12:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:12:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:12:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:12:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:12:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:12:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:12:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:12:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:12:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:12:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:13:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:13:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:13:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:13:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:13:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:13:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:13:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:13:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:13:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:13:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:13:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:14:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:14:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:14:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:14:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:14:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:14:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:14:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:14:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:14:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:14:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:14:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:14:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:15:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:15:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:15:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:15:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:15:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:15:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:15:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:15:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:15:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:15:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:15:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:16:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:16:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:16:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:16:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:16:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:16:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:16:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:16:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:16:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:16:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:16:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:16:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:17:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:17:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:17:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:17:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:17:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:17:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:17:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:17:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:17:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:17:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:17:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:18:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:18:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 15:18:11:[cuda:0] Epoch 800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 15:18:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 15:18:17:mr: 5971.607 INFO:Trainer:2024-05-23 15:18:17:mrr: 0.06 INFO:Trainer:2024-05-23 15:18:17:hits@1: 0.056 INFO:Trainer:2024-05-23 15:18:17:hits@3: 0.061 INFO:Trainer:2024-05-23 15:18:17:hits@10: 0.066 INFO:Trainer:2024-05-23 15:18:17:mr_type: 369.76 INFO:Trainer:2024-05-23 15:18:17:mrr_type: 0.153 INFO:Trainer:2024-05-23 15:18:17:hits@1_type: 0.114 INFO:Trainer:2024-05-23 15:18:17:hits@3_type: 0.153 INFO:Trainer:2024-05-23 15:18:17:hits@10_type: 0.222 INFO:EarlyStopping:2024-05-23 15:18:17:Validation score improved (0.062000 --> 0.066000). Saving model ... INFO:Trainer:2024-05-23 15:18:17:[cuda:0] Epoch [ 800/10000] | Batchsize: 4096 | loss: 0.101024 | 6.20122 seconds/epoch DEBUG:connectionpool:2024-05-23 15:18:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:18:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:18:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:18:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:18:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:18:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:18:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:18:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:19:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:19:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:19:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:19:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:19:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:19:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:19:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:19:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:19:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:19:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:19:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:19:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:20:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:20:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:20:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:20:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:20:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:20:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:20:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:20:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:20:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:20:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:20:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:21:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:21:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:21:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:21:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:21:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:21:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:21:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:21:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:21:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:21:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:21:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:22:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:22:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:22:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:22:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:22:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:22:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:22:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:22:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:22:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:22:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:22:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:22:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:23:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:23:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:23:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:23:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:23:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:23:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:23:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:23:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:23:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:23:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:23:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:24:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:24:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:24:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:24:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:24:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:24:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:24:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:24:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:24:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:24:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:24:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:24:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:25:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:25:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:25:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:25:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:25:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:25:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:25:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:25:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:25:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:25:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:25:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:26:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:26:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:26:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:26:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:26:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:26:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:26:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:26:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:26:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:26:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:26:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:27:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:27:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:27:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:27:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:27:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:27:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:27:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:27:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:27:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:27:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:27:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:27:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:28:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:28:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:28:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:28:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:28:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:28:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 15:28:32:[cuda:0] Epoch 900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 15:28:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 15:28:38:mr: 5887.539 INFO:Trainer:2024-05-23 15:28:38:mrr: 0.062 INFO:Trainer:2024-05-23 15:28:38:hits@1: 0.058 INFO:Trainer:2024-05-23 15:28:38:hits@3: 0.064 INFO:Trainer:2024-05-23 15:28:38:hits@10: 0.068 INFO:Trainer:2024-05-23 15:28:38:mr_type: 369.22 INFO:Trainer:2024-05-23 15:28:38:mrr_type: 0.157 INFO:Trainer:2024-05-23 15:28:38:hits@1_type: 0.117 INFO:Trainer:2024-05-23 15:28:38:hits@3_type: 0.157 INFO:Trainer:2024-05-23 15:28:38:hits@10_type: 0.227 INFO:EarlyStopping:2024-05-23 15:28:38:Validation score improved (0.066000 --> 0.068000). Saving model ... INFO:Trainer:2024-05-23 15:28:38:[cuda:0] Epoch [ 900/10000] | Batchsize: 4096 | loss: 0.099874 | 6.20222 seconds/epoch DEBUG:connectionpool:2024-05-23 15:28:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:28:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:28:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:28:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:29:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:29:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:29:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:29:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:29:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:29:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:29:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:29:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:29:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:29:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:29:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:29:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:30:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:30:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:30:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:30:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:30:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:30:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:30:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:30:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:30:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:30:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:30:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:31:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:31:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:31:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:31:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:31:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:31:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:31:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:31:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:31:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:31:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:31:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:32:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:32:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:32:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:32:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:32:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:32:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:32:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:32:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:32:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:32:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:32:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:32:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:33:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:33:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:33:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:33:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:33:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:33:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:33:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:33:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:33:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:33:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:33:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:34:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:34:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:34:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:34:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:34:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:34:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:34:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:34:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:34:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:34:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:34:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:34:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:35:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:35:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:35:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:35:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:35:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:35:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:35:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:35:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:35:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:35:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:35:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:36:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:36:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:36:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:36:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:36:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:36:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:36:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:36:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:36:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:36:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:36:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:37:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:37:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:37:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:37:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:37:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:37:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:37:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:37:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:37:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:37:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:37:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:37:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:38:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:38:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:38:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:38:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:38:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:38:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:38:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:38:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:38:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:38:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:38:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 15:38:59:[cuda:0] Epoch 1000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 15:39:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 15:39:04:mr: 5808.186 INFO:Trainer:2024-05-23 15:39:04:mrr: 0.065 INFO:Trainer:2024-05-23 15:39:04:hits@1: 0.06 INFO:Trainer:2024-05-23 15:39:04:hits@3: 0.066 INFO:Trainer:2024-05-23 15:39:04:hits@10: 0.071 INFO:Trainer:2024-05-23 15:39:04:mr_type: 368.436 INFO:Trainer:2024-05-23 15:39:04:mrr_type: 0.16 INFO:Trainer:2024-05-23 15:39:04:hits@1_type: 0.12 INFO:Trainer:2024-05-23 15:39:04:hits@3_type: 0.16 INFO:Trainer:2024-05-23 15:39:04:hits@10_type: 0.232 INFO:EarlyStopping:2024-05-23 15:39:04:Validation score improved (0.068000 --> 0.071000). Saving model ... INFO:Trainer:2024-05-23 15:39:05:[cuda:0] Epoch [1000/10000] | Batchsize: 4096 | loss: 0.098579 | 6.20898 seconds/epoch DEBUG:connectionpool:2024-05-23 15:39:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:39:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:39:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:39:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:39:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:39:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:39:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:39:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:39:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:39:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:40:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:40:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:40:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:40:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:40:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:40:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:40:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:40:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:40:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:40:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:40:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:40:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:41:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:41:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:41:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:41:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:41:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:41:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:41:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:41:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:41:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:41:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:41:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:42:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:42:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:42:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:42:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:42:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:42:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:42:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:42:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:42:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:42:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:42:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:42:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:43:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:43:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:43:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:43:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:43:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:43:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:43:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:43:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:43:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:43:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:43:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:44:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:44:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:44:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:44:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:44:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:44:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:44:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:44:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:44:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:44:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:44:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:45:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:45:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:45:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:45:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:45:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:45:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:45:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:45:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:45:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:45:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:45:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:45:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:46:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:46:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:46:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:46:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:46:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:46:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:46:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:46:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:46:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:46:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:46:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:47:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:47:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:47:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:47:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:47:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:47:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:47:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:47:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:47:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:47:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:47:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:48:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:48:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:48:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:48:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:48:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:48:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:48:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:48:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:48:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:48:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:48:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:48:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:49:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:49:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:49:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 15:49:18:[cuda:0] Epoch 1100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 15:49:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 15:49:24:mr: 5737.169 INFO:Trainer:2024-05-23 15:49:24:mrr: 0.067 INFO:Trainer:2024-05-23 15:49:24:hits@1: 0.062 INFO:Trainer:2024-05-23 15:49:24:hits@3: 0.068 INFO:Trainer:2024-05-23 15:49:24:hits@10: 0.075 INFO:Trainer:2024-05-23 15:49:24:mr_type: 367.906 INFO:Trainer:2024-05-23 15:49:24:mrr_type: 0.164 INFO:Trainer:2024-05-23 15:49:24:hits@1_type: 0.123 INFO:Trainer:2024-05-23 15:49:24:hits@3_type: 0.164 INFO:Trainer:2024-05-23 15:49:24:hits@10_type: 0.236 INFO:EarlyStopping:2024-05-23 15:49:24:Validation score improved (0.071000 --> 0.075000). Saving model ... INFO:Trainer:2024-05-23 15:49:24:[cuda:0] Epoch [1100/10000] | Batchsize: 4096 | loss: 0.097200 | 6.20766 seconds/epoch DEBUG:connectionpool:2024-05-23 15:49:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:49:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:49:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:49:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:49:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:49:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:49:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:50:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:50:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:50:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:50:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:50:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:50:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:50:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:50:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:50:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:50:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:50:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:50:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:51:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:51:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:51:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:51:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:51:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:51:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:51:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:51:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:51:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:51:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:51:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:52:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:52:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:52:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:52:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:52:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:52:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:52:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:52:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:52:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:52:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:52:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:53:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:53:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:53:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:53:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:53:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:53:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:53:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:53:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:53:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:53:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:53:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:53:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:54:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:54:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:54:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:54:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:54:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:54:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:54:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:54:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:54:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:54:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:54:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:55:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:55:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:55:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:55:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:55:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:55:27:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-23 15:55:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:55:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:55:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:55:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:55:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:55:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:55:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:56:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:56:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:56:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:56:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:56:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:56:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:56:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:56:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:56:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:56:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:56:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:57:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:57:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:57:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:57:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:57:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:57:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:57:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:57:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:57:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:57:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:57:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:58:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:58:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:58:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:58:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:58:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:58:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:58:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:58:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:58:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:58:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:58:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:58:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:59:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:59:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:59:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:59:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:59:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:59:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:59:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 15:59:40:[cuda:0] Epoch 1200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 15:59:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 15:59:45:mr: 5669.333 INFO:Trainer:2024-05-23 15:59:45:mrr: 0.07 INFO:Trainer:2024-05-23 15:59:45:hits@1: 0.064 INFO:Trainer:2024-05-23 15:59:45:hits@3: 0.071 INFO:Trainer:2024-05-23 15:59:45:hits@10: 0.08 INFO:Trainer:2024-05-23 15:59:45:mr_type: 367.263 INFO:Trainer:2024-05-23 15:59:45:mrr_type: 0.166 INFO:Trainer:2024-05-23 15:59:45:hits@1_type: 0.125 INFO:Trainer:2024-05-23 15:59:45:hits@3_type: 0.168 INFO:Trainer:2024-05-23 15:59:45:hits@10_type: 0.24 INFO:EarlyStopping:2024-05-23 15:59:45:Validation score improved (0.075000 --> 0.080000). Saving model ... INFO:Trainer:2024-05-23 15:59:46:[cuda:0] Epoch [1200/10000] | Batchsize: 4096 | loss: 0.096332 | 6.20859 seconds/epoch DEBUG:connectionpool:2024-05-23 15:59:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:59:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 15:59:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:00:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:00:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:00:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:00:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:00:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:00:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:00:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:00:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:00:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:00:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:00:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:01:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:01:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:01:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:01:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:01:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:01:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:01:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:01:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:01:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:01:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:01:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:01:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:02:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:02:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:02:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:02:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:02:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:02:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:02:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:02:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:02:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:02:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:02:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:03:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:03:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:03:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:03:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:03:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:03:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:03:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:03:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:03:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:03:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:03:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:04:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:04:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:04:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:04:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:04:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:04:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:04:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:04:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:04:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:04:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:04:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:04:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:05:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:05:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:05:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:05:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:05:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:05:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:05:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:05:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:05:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:05:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:05:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:06:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:06:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:06:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:06:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:06:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:06:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:06:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:06:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:06:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:06:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:06:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:07:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:07:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:07:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:07:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:07:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:07:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:07:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:07:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:07:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:07:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:07:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:07:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:08:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:08:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:08:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:08:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:08:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:08:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:08:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:08:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:08:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:08:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:08:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:09:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:09:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:09:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:09:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:09:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:09:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:09:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:09:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:09:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:09:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:09:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:09:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 16:10:03:[cuda:0] Epoch 1300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 16:10:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 16:10:09:mr: 5605.491 INFO:Trainer:2024-05-23 16:10:09:mrr: 0.073 INFO:Trainer:2024-05-23 16:10:09:hits@1: 0.065 INFO:Trainer:2024-05-23 16:10:09:hits@3: 0.074 INFO:Trainer:2024-05-23 16:10:09:hits@10: 0.085 INFO:Trainer:2024-05-23 16:10:09:mr_type: 366.633 INFO:Trainer:2024-05-23 16:10:09:mrr_type: 0.169 INFO:Trainer:2024-05-23 16:10:09:hits@1_type: 0.128 INFO:Trainer:2024-05-23 16:10:09:hits@3_type: 0.171 INFO:Trainer:2024-05-23 16:10:09:hits@10_type: 0.244 INFO:EarlyStopping:2024-05-23 16:10:09:Validation score improved (0.080000 --> 0.085000). Saving model ... INFO:Trainer:2024-05-23 16:10:09:[cuda:0] Epoch [1300/10000] | Batchsize: 4096 | loss: 0.095608 | 6.21026 seconds/epoch DEBUG:connectionpool:2024-05-23 16:10:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:10:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:10:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:10:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:10:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:10:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:10:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:10:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:10:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:10:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:11:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:11:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:11:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:11:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:11:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:11:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:11:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:11:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:11:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:11:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:11:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:12:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:12:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:12:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:12:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:12:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:12:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:12:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:12:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:12:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:12:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:12:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:12:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:13:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:13:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:13:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:13:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:13:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:13:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:13:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:13:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:13:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:13:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:13:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:14:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:14:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:14:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:14:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:14:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:14:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:14:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:14:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:14:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:14:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:14:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:15:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:15:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:15:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:15:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:15:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:15:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:15:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:15:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:15:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:15:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:15:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:15:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:16:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:16:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:16:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:16:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:16:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:16:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:16:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:16:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:16:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:16:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:16:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:17:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:17:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:17:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:17:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:17:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:17:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:17:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:17:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:17:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:17:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:17:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:18:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:18:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:18:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:18:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:18:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:18:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:18:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:18:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:18:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:18:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:18:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:19:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:19:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:19:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:19:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:19:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:19:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:19:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:19:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:19:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:19:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:19:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:19:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:20:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:20:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:20:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:20:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:20:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:20:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 16:20:31:[cuda:0] Epoch 1400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 16:20:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 16:20:38:mr: 5546.424 INFO:Trainer:2024-05-23 16:20:38:mrr: 0.075 INFO:Trainer:2024-05-23 16:20:38:hits@1: 0.066 INFO:Trainer:2024-05-23 16:20:38:hits@3: 0.078 INFO:Trainer:2024-05-23 16:20:38:hits@10: 0.09 INFO:Trainer:2024-05-23 16:20:38:mr_type: 365.995 INFO:Trainer:2024-05-23 16:20:38:mrr_type: 0.172 INFO:Trainer:2024-05-23 16:20:38:hits@1_type: 0.13 INFO:Trainer:2024-05-23 16:20:38:hits@3_type: 0.175 INFO:Trainer:2024-05-23 16:20:38:hits@10_type: 0.248 INFO:EarlyStopping:2024-05-23 16:20:38:Validation score improved (0.085000 --> 0.090000). Saving model ... INFO:Trainer:2024-05-23 16:20:38:[cuda:0] Epoch [1400/10000] | Batchsize: 4096 | loss: 0.094825 | 6.21559 seconds/epoch DEBUG:connectionpool:2024-05-23 16:20:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:20:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:20:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:20:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:21:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:21:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:21:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:21:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:21:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:21:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:21:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:21:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:21:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:21:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:21:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:22:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:22:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:22:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:22:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:22:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:22:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:22:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:22:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:22:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:22:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:22:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:22:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:23:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:23:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:23:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:23:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:23:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:23:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:23:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:23:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:23:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:23:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:23:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:24:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:24:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:24:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:24:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:24:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:24:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:24:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:24:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:24:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:24:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:24:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:25:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:25:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:25:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:25:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:25:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:25:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:25:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:25:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:25:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:25:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:25:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:25:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:26:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:26:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:26:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:26:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:26:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:26:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:26:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:26:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:26:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:26:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:26:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:27:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:27:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:27:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:27:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:27:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:27:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:27:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:27:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:27:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:27:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:27:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:27:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:28:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:28:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:28:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:28:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:28:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:28:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:28:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:28:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:28:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:28:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:28:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:29:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:29:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:29:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:29:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:29:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:29:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:29:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:29:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:29:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:29:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:29:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:30:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:30:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:30:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:30:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:30:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:30:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:30:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:30:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:30:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:30:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:30:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 16:30:57:[cuda:0] Epoch 1500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 16:30:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 16:31:03:mr: 5487.385 INFO:Trainer:2024-05-23 16:31:03:mrr: 0.077 INFO:Trainer:2024-05-23 16:31:03:hits@1: 0.067 INFO:Trainer:2024-05-23 16:31:03:hits@3: 0.081 INFO:Trainer:2024-05-23 16:31:03:hits@10: 0.094 INFO:Trainer:2024-05-23 16:31:03:mr_type: 365.139 INFO:Trainer:2024-05-23 16:31:03:mrr_type: 0.175 INFO:Trainer:2024-05-23 16:31:03:hits@1_type: 0.132 INFO:Trainer:2024-05-23 16:31:03:hits@3_type: 0.178 INFO:Trainer:2024-05-23 16:31:03:hits@10_type: 0.251 INFO:EarlyStopping:2024-05-23 16:31:03:Validation score improved (0.090000 --> 0.094000). Saving model ... INFO:Trainer:2024-05-23 16:31:03:[cuda:0] Epoch [1500/10000] | Batchsize: 4096 | loss: 0.094151 | 6.21840 seconds/epoch DEBUG:connectionpool:2024-05-23 16:31:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:31:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:31:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:31:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:31:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:31:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:31:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:31:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:31:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:31:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:31:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:32:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:32:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:32:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:32:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:32:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:32:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:32:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:32:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:32:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:32:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:32:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:33:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:33:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:33:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:33:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:33:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:33:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:33:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:33:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:33:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:33:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:33:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:33:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:34:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:34:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:34:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:34:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:34:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:34:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:34:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:34:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:34:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:34:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:34:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:35:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:35:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:35:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:35:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:35:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:35:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:35:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:35:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:35:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:35:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:35:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:36:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:36:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:36:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:36:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:36:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:36:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:36:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:36:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:36:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:36:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:36:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:36:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:37:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:37:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:37:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:37:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:37:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:37:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:37:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:37:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:37:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:37:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:37:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:38:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:38:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:38:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:38:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:38:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:38:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:38:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:38:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:38:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:38:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:38:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:38:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:39:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:39:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:39:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:39:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:39:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:39:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:39:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:39:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:39:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:39:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:39:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:40:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:40:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:40:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:40:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:40:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:40:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:40:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:40:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:40:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:40:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:40:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:41:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:41:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:41:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:41:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:41:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 16:41:23:[cuda:0] Epoch 1600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 16:41:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 16:41:29:mr: 5430.614 INFO:Trainer:2024-05-23 16:41:29:mrr: 0.079 INFO:Trainer:2024-05-23 16:41:29:hits@1: 0.068 INFO:Trainer:2024-05-23 16:41:29:hits@3: 0.084 INFO:Trainer:2024-05-23 16:41:29:hits@10: 0.098 INFO:Trainer:2024-05-23 16:41:29:mr_type: 364.268 INFO:Trainer:2024-05-23 16:41:29:mrr_type: 0.177 INFO:Trainer:2024-05-23 16:41:29:hits@1_type: 0.134 INFO:Trainer:2024-05-23 16:41:29:hits@3_type: 0.18 INFO:Trainer:2024-05-23 16:41:29:hits@10_type: 0.255 INFO:EarlyStopping:2024-05-23 16:41:29:Validation score improved (0.094000 --> 0.098000). Saving model ... INFO:Trainer:2024-05-23 16:41:30:[cuda:0] Epoch [1600/10000] | Batchsize: 4096 | loss: 0.093416 | 6.22104 seconds/epoch DEBUG:connectionpool:2024-05-23 16:41:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:41:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:41:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:41:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:41:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:41:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:42:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:42:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:42:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:42:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:42:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:42:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:42:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:42:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:42:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:42:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:42:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:43:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:43:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:43:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:43:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:43:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:43:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:43:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:43:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:43:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:43:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:43:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:44:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:44:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:44:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:44:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:44:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:44:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:44:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:44:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:44:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:44:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:44:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:44:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:45:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:45:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:45:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:45:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:45:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:45:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:45:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:45:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:45:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:45:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:45:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:46:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:46:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:46:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:46:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:46:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:46:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:46:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:46:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:46:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:46:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:46:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:47:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:47:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:47:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:47:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:47:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:47:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:47:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:47:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:47:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:47:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:47:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:47:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:48:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:48:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:48:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:48:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:48:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:48:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:48:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:48:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:48:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:48:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:48:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:49:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:49:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:49:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:49:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:49:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:49:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:49:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:49:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:49:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:49:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:49:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:49:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:50:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:50:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:50:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:50:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:50:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:50:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:50:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:50:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:50:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:50:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:50:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:51:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:51:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:51:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:51:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:51:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:51:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:51:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:51:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:51:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:51:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 16:51:50:[cuda:0] Epoch 1700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 16:51:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 16:51:57:mr: 5372.73 INFO:Trainer:2024-05-23 16:51:57:mrr: 0.08 INFO:Trainer:2024-05-23 16:51:57:hits@1: 0.069 INFO:Trainer:2024-05-23 16:51:57:hits@3: 0.086 INFO:Trainer:2024-05-23 16:51:57:hits@10: 0.1 INFO:Trainer:2024-05-23 16:51:57:mr_type: 363.987 INFO:Trainer:2024-05-23 16:51:57:mrr_type: 0.179 INFO:Trainer:2024-05-23 16:51:57:hits@1_type: 0.135 INFO:Trainer:2024-05-23 16:51:57:hits@3_type: 0.182 INFO:Trainer:2024-05-23 16:51:57:hits@10_type: 0.259 INFO:EarlyStopping:2024-05-23 16:51:57:Validation score improved (0.098000 --> 0.100000). Saving model ... INFO:Trainer:2024-05-23 16:51:57:[cuda:0] Epoch [1700/10000] | Batchsize: 4096 | loss: 0.092964 | 6.22405 seconds/epoch DEBUG:connectionpool:2024-05-23 16:52:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:52:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:52:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:52:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:52:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:52:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:52:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:52:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:52:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:52:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:52:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:52:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:53:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:53:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:53:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:53:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:53:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:53:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:53:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:53:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:53:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:53:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:53:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:54:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:54:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:54:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:54:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:54:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:54:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:54:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:54:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:54:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:54:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:54:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:55:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:55:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:55:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:55:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:55:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:55:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:55:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:55:37:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-23 16:55:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:55:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:55:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:55:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:55:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:56:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:56:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:56:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:56:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:56:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:56:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:56:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:56:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:56:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:56:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:56:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:57:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:57:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:57:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:57:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:57:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:57:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:57:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:57:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:57:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:57:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:57:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:58:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:58:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:58:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:58:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:58:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:58:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:58:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:58:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:58:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:58:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:58:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:58:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:59:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:59:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:59:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:59:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:59:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:59:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:59:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:59:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:59:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:59:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 16:59:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:00:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:00:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:00:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:00:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:00:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:00:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:00:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:00:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:00:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:00:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:00:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:01:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:01:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:01:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:01:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:01:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:01:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:01:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:01:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:01:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:01:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:01:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:01:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:02:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:02:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:02:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:02:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 17:02:21:[cuda:0] Epoch 1800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 17:02:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 17:02:27:mr: 5320.551 INFO:Trainer:2024-05-23 17:02:27:mrr: 0.082 INFO:Trainer:2024-05-23 17:02:27:hits@1: 0.07 INFO:Trainer:2024-05-23 17:02:27:hits@3: 0.088 INFO:Trainer:2024-05-23 17:02:27:hits@10: 0.103 INFO:Trainer:2024-05-23 17:02:27:mr_type: 363.426 INFO:Trainer:2024-05-23 17:02:27:mrr_type: 0.181 INFO:Trainer:2024-05-23 17:02:27:hits@1_type: 0.136 INFO:Trainer:2024-05-23 17:02:27:hits@3_type: 0.185 INFO:Trainer:2024-05-23 17:02:27:hits@10_type: 0.264 INFO:EarlyStopping:2024-05-23 17:02:27:Validation score improved (0.100000 --> 0.103000). Saving model ... INFO:Trainer:2024-05-23 17:02:28:[cuda:0] Epoch [1800/10000] | Batchsize: 4096 | loss: 0.092404 | 6.22875 seconds/epoch DEBUG:connectionpool:2024-05-23 17:02:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:02:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:02:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:02:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:02:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:02:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:03:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:03:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:03:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:03:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:03:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:03:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:03:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:03:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:03:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:03:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:03:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:04:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:04:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:04:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:04:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:04:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:04:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:04:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:04:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:04:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:04:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:04:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:04:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:05:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:05:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:05:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:05:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:05:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:05:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:05:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:05:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:05:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:05:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:05:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:06:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:06:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:06:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:06:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:06:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:06:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:06:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:06:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:06:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:06:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:06:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:07:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:07:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:07:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:07:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:07:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:07:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:07:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:07:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:07:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:07:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:07:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:07:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:08:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:08:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:08:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:08:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:08:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:08:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:08:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:08:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:08:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:08:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:08:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:09:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:09:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:09:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:09:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:09:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:09:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:09:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:09:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:09:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:09:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:09:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:10:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:10:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:10:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:10:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:10:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:10:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:10:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:10:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:10:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:10:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:10:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:10:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:11:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:11:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:11:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:11:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:11:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:11:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:11:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:11:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:11:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:11:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:11:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:12:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:12:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:12:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:12:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:12:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:12:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:12:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:12:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:12:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:12:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 17:12:52:[cuda:0] Epoch 1900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 17:12:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 17:12:58:mr: 5267.876 INFO:Trainer:2024-05-23 17:12:58:mrr: 0.083 INFO:Trainer:2024-05-23 17:12:58:hits@1: 0.07 INFO:Trainer:2024-05-23 17:12:58:hits@3: 0.089 INFO:Trainer:2024-05-23 17:12:58:hits@10: 0.106 INFO:Trainer:2024-05-23 17:12:58:mr_type: 362.735 INFO:Trainer:2024-05-23 17:12:58:mrr_type: 0.183 INFO:Trainer:2024-05-23 17:12:58:hits@1_type: 0.137 INFO:Trainer:2024-05-23 17:12:58:hits@3_type: 0.188 INFO:Trainer:2024-05-23 17:12:58:hits@10_type: 0.266 INFO:EarlyStopping:2024-05-23 17:12:58:Validation score improved (0.103000 --> 0.106000). Saving model ... INFO:Trainer:2024-05-23 17:12:58:[cuda:0] Epoch [1900/10000] | Batchsize: 4096 | loss: 0.091842 | 6.23273 seconds/epoch DEBUG:connectionpool:2024-05-23 17:13:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:13:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:13:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:13:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:13:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:13:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:13:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:13:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:13:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:13:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:13:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:13:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:14:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:14:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:14:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:14:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:14:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:14:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:14:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:14:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:14:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:14:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:14:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:15:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:15:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:15:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:15:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:15:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:15:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:15:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:15:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:15:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:15:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:15:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:16:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:16:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:16:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:16:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:16:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:16:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:16:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:16:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:16:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:16:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:16:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:17:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:17:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:17:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:17:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:17:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:17:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:17:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:17:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:17:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:17:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:17:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:17:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:18:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:18:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:18:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:18:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:18:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:18:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:18:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:18:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:18:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:18:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:18:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:19:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:19:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:19:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:19:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:19:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:19:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:19:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:19:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:19:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:19:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:19:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:19:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:20:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:20:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:20:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:20:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:20:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:20:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:20:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:20:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:20:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:20:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:20:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:21:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:21:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:21:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:21:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:21:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:21:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:21:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:21:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:21:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:21:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:21:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:22:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:22:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:22:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:22:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:22:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:22:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:22:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:22:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:22:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:22:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:22:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:23:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:23:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:23:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:23:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:23:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:23:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 17:23:26:[cuda:0] Epoch 2000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 17:23:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 17:23:32:mr: 5216.729 INFO:Trainer:2024-05-23 17:23:32:mrr: 0.085 INFO:Trainer:2024-05-23 17:23:32:hits@1: 0.071 INFO:Trainer:2024-05-23 17:23:32:hits@3: 0.091 INFO:Trainer:2024-05-23 17:23:32:hits@10: 0.11 INFO:Trainer:2024-05-23 17:23:32:mr_type: 362.117 INFO:Trainer:2024-05-23 17:23:32:mrr_type: 0.185 INFO:Trainer:2024-05-23 17:23:32:hits@1_type: 0.139 INFO:Trainer:2024-05-23 17:23:32:hits@3_type: 0.191 INFO:Trainer:2024-05-23 17:23:32:hits@10_type: 0.269 INFO:EarlyStopping:2024-05-23 17:23:32:Validation score improved (0.106000 --> 0.110000). Saving model ... INFO:Trainer:2024-05-23 17:23:32:[cuda:0] Epoch [2000/10000] | Batchsize: 4096 | loss: 0.091416 | 6.23809 seconds/epoch DEBUG:connectionpool:2024-05-23 17:23:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:23:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:23:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:23:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:23:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:24:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:24:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:24:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:24:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:24:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:24:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:24:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:24:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:24:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:24:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:24:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:25:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:25:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:25:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:25:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:25:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:25:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:25:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:25:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:25:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:25:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:25:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:25:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:26:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:26:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:26:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:26:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:26:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:26:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:26:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:26:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:26:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:26:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:26:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:27:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:27:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:27:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:27:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:27:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:27:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:27:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:27:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:27:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:27:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:27:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:28:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:28:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:28:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:28:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:28:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:28:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:28:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:28:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:28:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:28:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:28:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:29:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:29:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:29:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:29:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:29:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:29:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:29:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:29:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:29:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:29:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:29:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:29:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:30:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:30:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:30:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:30:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:30:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:30:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:30:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:30:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:30:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:30:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:30:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:31:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:31:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:31:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:31:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:31:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:31:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:31:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:31:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:31:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:31:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:31:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:31:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:32:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:32:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:32:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:32:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:32:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:32:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:32:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:32:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:32:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:32:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:32:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:33:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:33:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:33:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:33:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:33:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:33:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:33:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:33:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:33:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:33:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:33:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 17:33:59:[cuda:0] Epoch 2100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 17:34:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 17:34:05:mr: 5171.169 INFO:Trainer:2024-05-23 17:34:05:mrr: 0.086 INFO:Trainer:2024-05-23 17:34:05:hits@1: 0.072 INFO:Trainer:2024-05-23 17:34:05:hits@3: 0.092 INFO:Trainer:2024-05-23 17:34:05:hits@10: 0.113 INFO:Trainer:2024-05-23 17:34:05:mr_type: 361.45 INFO:Trainer:2024-05-23 17:34:05:mrr_type: 0.186 INFO:Trainer:2024-05-23 17:34:05:hits@1_type: 0.14 INFO:Trainer:2024-05-23 17:34:05:hits@3_type: 0.192 INFO:Trainer:2024-05-23 17:34:05:hits@10_type: 0.271 INFO:EarlyStopping:2024-05-23 17:34:05:Validation score improved (0.110000 --> 0.113000). Saving model ... INFO:Trainer:2024-05-23 17:34:06:[cuda:0] Epoch [2100/10000] | Batchsize: 4096 | loss: 0.091078 | 6.24280 seconds/epoch DEBUG:connectionpool:2024-05-23 17:34:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:34:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:34:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:34:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:34:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:34:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:34:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:34:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:34:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:34:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:34:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:35:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:35:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:35:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:35:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:35:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:35:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:35:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:35:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:35:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:35:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:35:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:36:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:36:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:36:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:36:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:36:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:36:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:36:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:36:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:36:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:36:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:36:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:37:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:37:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:37:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:37:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:37:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:37:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:37:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:37:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:37:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:37:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:37:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:37:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:38:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:38:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:38:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:38:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:38:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:38:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:38:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:38:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:38:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:38:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:38:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:39:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:39:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:39:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:39:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:39:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:39:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:39:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:39:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:39:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:39:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:39:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:40:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:40:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:40:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:40:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:40:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:40:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:40:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:40:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:40:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:40:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:40:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:40:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:41:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:41:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:41:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:41:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:41:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:41:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:41:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:41:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:41:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:41:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:41:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:42:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:42:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:42:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:42:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:42:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:42:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:42:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:42:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:42:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:42:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:42:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:43:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:43:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:43:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:43:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:43:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:43:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:43:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:43:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:43:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:43:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:43:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:43:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:44:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:44:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:44:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:44:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:44:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:44:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:44:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 17:44:38:[cuda:0] Epoch 2200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 17:44:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 17:44:44:mr: 5119.574 INFO:Trainer:2024-05-23 17:44:44:mrr: 0.088 INFO:Trainer:2024-05-23 17:44:44:hits@1: 0.073 INFO:Trainer:2024-05-23 17:44:44:hits@3: 0.093 INFO:Trainer:2024-05-23 17:44:44:hits@10: 0.116 INFO:Trainer:2024-05-23 17:44:44:mr_type: 360.925 INFO:Trainer:2024-05-23 17:44:44:mrr_type: 0.188 INFO:Trainer:2024-05-23 17:44:44:hits@1_type: 0.141 INFO:Trainer:2024-05-23 17:44:44:hits@3_type: 0.195 INFO:Trainer:2024-05-23 17:44:44:hits@10_type: 0.274 INFO:EarlyStopping:2024-05-23 17:44:44:Validation score improved (0.113000 --> 0.116000). Saving model ... INFO:Trainer:2024-05-23 17:44:45:[cuda:0] Epoch [2200/10000] | Batchsize: 4096 | loss: 0.090769 | 6.24939 seconds/epoch DEBUG:connectionpool:2024-05-23 17:44:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:44:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:44:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:45:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:45:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:45:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:45:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:45:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:45:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:45:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:45:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:45:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:45:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:45:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:46:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:46:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:46:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:46:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:46:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:46:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:46:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:46:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:46:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:46:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:46:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:46:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:47:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:47:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:47:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:47:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:47:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:47:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:47:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:47:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:47:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:47:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:47:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:48:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:48:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:48:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:48:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:48:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:48:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:48:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:48:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:48:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:48:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:48:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:49:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:49:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:49:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:49:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:49:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:49:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:49:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:49:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:49:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:49:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:49:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:49:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:50:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:50:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:50:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:50:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:50:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:50:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:50:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:50:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:50:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:50:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:50:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:51:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:51:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:51:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:51:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:51:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:51:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:51:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:51:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:51:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:51:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:51:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:52:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:52:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:52:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:52:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:52:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:52:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:52:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:52:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:52:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:52:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:52:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:52:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:53:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:53:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:53:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:53:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:53:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:53:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:53:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:53:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:53:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:53:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:53:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:54:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:54:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:54:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:54:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:54:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:54:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:54:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:54:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:54:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:54:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:54:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:55:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:55:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 17:55:10:[cuda:0] Epoch 2300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 17:55:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:55:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 17:55:16:mr: 5065.032 INFO:Trainer:2024-05-23 17:55:16:mrr: 0.089 INFO:Trainer:2024-05-23 17:55:16:hits@1: 0.074 INFO:Trainer:2024-05-23 17:55:16:hits@3: 0.094 INFO:Trainer:2024-05-23 17:55:16:hits@10: 0.119 INFO:Trainer:2024-05-23 17:55:16:mr_type: 359.87 INFO:Trainer:2024-05-23 17:55:16:mrr_type: 0.19 INFO:Trainer:2024-05-23 17:55:16:hits@1_type: 0.142 INFO:Trainer:2024-05-23 17:55:16:hits@3_type: 0.196 INFO:Trainer:2024-05-23 17:55:16:hits@10_type: 0.277 INFO:EarlyStopping:2024-05-23 17:55:16:Validation score improved (0.116000 --> 0.119000). Saving model ... INFO:Trainer:2024-05-23 17:55:16:[cuda:0] Epoch [2300/10000] | Batchsize: 4096 | loss: 0.090282 | 6.25240 seconds/epoch DEBUG:connectionpool:2024-05-23 17:55:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:55:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:55:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:55:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:55:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:55:47:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-23 17:55:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:55:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:55:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:56:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:56:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:56:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:56:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:56:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:56:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:56:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:56:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:56:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:56:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:56:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:57:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:57:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:57:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:57:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:57:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:57:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:57:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:57:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:57:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:57:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:57:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:58:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:58:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:58:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:58:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:58:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:58:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:58:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:58:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:58:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:58:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:58:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:58:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:59:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:59:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:59:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:59:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:59:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:59:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:59:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:59:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:59:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:59:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 17:59:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:00:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:00:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:00:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:00:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:00:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:00:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:00:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:00:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:00:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:00:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:00:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:01:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:01:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:01:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:01:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:01:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:01:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:01:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:01:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:01:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:01:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:01:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:01:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:02:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:02:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:02:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:02:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:02:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:02:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:02:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:02:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:02:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:02:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:02:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:03:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:03:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:03:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:03:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:03:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:03:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:03:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:03:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:03:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:03:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:03:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:04:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:04:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:04:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:04:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:04:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:04:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:04:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:04:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:04:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:04:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:04:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:04:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:05:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:05:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:05:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:05:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:05:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:05:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:05:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:05:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:05:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:05:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:05:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:05:58:[cuda:0] Epoch 2400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 18:06:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:06:04:mr: 5025.033 INFO:Trainer:2024-05-23 18:06:04:mrr: 0.091 INFO:Trainer:2024-05-23 18:06:04:hits@1: 0.076 INFO:Trainer:2024-05-23 18:06:04:hits@3: 0.096 INFO:Trainer:2024-05-23 18:06:04:hits@10: 0.122 INFO:Trainer:2024-05-23 18:06:04:mr_type: 359.274 INFO:Trainer:2024-05-23 18:06:04:mrr_type: 0.191 INFO:Trainer:2024-05-23 18:06:04:hits@1_type: 0.143 INFO:Trainer:2024-05-23 18:06:04:hits@3_type: 0.198 INFO:Trainer:2024-05-23 18:06:04:hits@10_type: 0.279 INFO:EarlyStopping:2024-05-23 18:06:04:Validation score improved (0.119000 --> 0.122000). Saving model ... INFO:Trainer:2024-05-23 18:06:05:[cuda:0] Epoch [2400/10000] | Batchsize: 4096 | loss: 0.090261 | 6.26194 seconds/epoch DEBUG:connectionpool:2024-05-23 18:06:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:06:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:06:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:06:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:06:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:06:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:06:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:06:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:06:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:06:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:07:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:07:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:07:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:07:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:07:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:07:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:07:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:07:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:07:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:07:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:07:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:07:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:08:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:08:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:08:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:08:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:08:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:08:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:08:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:08:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:08:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:08:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:08:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:09:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:09:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:09:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:09:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:09:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:09:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:09:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:09:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:09:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:09:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:09:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:10:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:10:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:10:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:10:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:10:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:10:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:10:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:10:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:10:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:10:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:10:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:10:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:11:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:11:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:11:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:11:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:11:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:11:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:11:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:11:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:11:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:11:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:11:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:12:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:12:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:12:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:12:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:12:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:12:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:12:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:12:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:12:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:12:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:12:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:12:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:13:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:13:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:13:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:13:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:13:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:13:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:13:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:13:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:13:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:13:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:13:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:14:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:14:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:14:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:14:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:14:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:14:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:14:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:14:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:14:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:14:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:14:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:15:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:15:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:15:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:15:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:15:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:15:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:15:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:15:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:15:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:15:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:15:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:15:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:16:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:16:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:16:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:16:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:16:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:16:30:[cuda:0] Epoch 2500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 18:16:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:16:36:mr: 4982.559 INFO:Trainer:2024-05-23 18:16:36:mrr: 0.093 INFO:Trainer:2024-05-23 18:16:36:hits@1: 0.077 INFO:Trainer:2024-05-23 18:16:36:hits@3: 0.098 INFO:Trainer:2024-05-23 18:16:36:hits@10: 0.125 INFO:Trainer:2024-05-23 18:16:36:mr_type: 358.743 INFO:Trainer:2024-05-23 18:16:36:mrr_type: 0.193 INFO:Trainer:2024-05-23 18:16:36:hits@1_type: 0.144 INFO:Trainer:2024-05-23 18:16:36:hits@3_type: 0.2 INFO:Trainer:2024-05-23 18:16:36:hits@10_type: 0.282 INFO:EarlyStopping:2024-05-23 18:16:36:Validation score improved (0.122000 --> 0.125000). Saving model ... DEBUG:connectionpool:2024-05-23 18:16:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:16:36:[cuda:0] Epoch [2500/10000] | Batchsize: 4096 | loss: 0.089790 | 6.26411 seconds/epoch DEBUG:connectionpool:2024-05-23 18:16:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:16:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:16:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:16:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:17:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:17:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:17:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:17:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:17:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:17:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:17:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:17:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:17:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:17:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:17:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:18:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:18:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:18:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:18:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:18:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:18:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:18:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:18:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:18:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:18:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:18:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:18:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:19:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:19:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:19:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:19:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:19:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:19:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:19:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:19:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:19:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:19:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:19:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:20:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:20:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:20:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:20:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:20:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:20:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:20:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:20:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:20:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:20:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:20:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:21:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:21:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:21:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:21:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:21:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:21:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:21:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:21:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:21:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:21:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:21:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:21:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:22:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:22:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:22:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:22:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:22:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:22:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:22:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:22:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:22:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:22:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:22:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:23:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:23:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:23:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:23:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:23:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:23:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:23:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:23:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:23:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:23:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:23:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:23:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:24:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:24:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:24:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:24:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:24:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:24:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:24:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:24:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:24:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:24:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:24:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:25:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:25:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:25:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:25:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:25:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:25:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:25:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:25:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:25:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:25:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:25:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:26:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:26:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:26:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:26:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:26:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:26:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:26:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:26:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:26:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:26:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:26:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:26:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:27:04:[cuda:0] Epoch 2600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 18:27:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:27:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:27:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:27:16:mr: 4939.877 INFO:Trainer:2024-05-23 18:27:16:mrr: 0.095 INFO:Trainer:2024-05-23 18:27:16:hits@1: 0.078 INFO:Trainer:2024-05-23 18:27:16:hits@3: 0.099 INFO:Trainer:2024-05-23 18:27:16:hits@10: 0.127 INFO:Trainer:2024-05-23 18:27:16:mr_type: 358.227 INFO:Trainer:2024-05-23 18:27:16:mrr_type: 0.194 INFO:Trainer:2024-05-23 18:27:16:hits@1_type: 0.145 INFO:Trainer:2024-05-23 18:27:16:hits@3_type: 0.201 INFO:Trainer:2024-05-23 18:27:16:hits@10_type: 0.285 INFO:EarlyStopping:2024-05-23 18:27:16:Validation score improved (0.125000 --> 0.127000). Saving model ... INFO:Trainer:2024-05-23 18:27:16:[cuda:0] Epoch [2600/10000] | Batchsize: 4096 | loss: 0.089631 | 6.26708 seconds/epoch DEBUG:connectionpool:2024-05-23 18:27:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:27:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:27:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:27:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:27:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:27:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:27:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:27:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:28:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:28:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:28:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:28:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:28:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:28:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:28:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:28:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:28:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:28:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:28:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:29:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:29:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:29:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:29:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:29:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:29:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:29:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:29:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:29:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:29:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:29:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:29:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:30:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:30:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:30:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:30:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:30:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:30:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:30:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:30:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:30:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:30:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:30:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:31:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:31:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:31:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:31:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:31:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:31:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:31:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:31:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:31:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:31:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:31:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:32:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:32:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:32:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:32:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:32:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:32:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:32:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:32:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:32:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:32:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:32:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:32:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:33:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:33:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:33:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:33:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:33:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:33:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:33:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:33:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:33:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:33:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:33:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:34:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:34:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:34:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:34:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:34:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:34:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:34:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:34:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:34:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:34:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:34:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:35:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:35:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:35:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:35:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:35:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:35:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:35:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:35:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:35:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:35:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:35:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:35:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:36:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:36:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:36:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:36:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:36:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:36:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:36:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:36:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:36:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:36:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:36:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:37:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:37:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:37:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:37:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:37:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:37:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:37:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:37:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:37:44:[cuda:0] Epoch 2700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 18:37:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:37:49:mr: 4890.436 INFO:Trainer:2024-05-23 18:37:49:mrr: 0.096 INFO:Trainer:2024-05-23 18:37:49:hits@1: 0.079 INFO:Trainer:2024-05-23 18:37:49:hits@3: 0.101 INFO:Trainer:2024-05-23 18:37:49:hits@10: 0.13 INFO:Trainer:2024-05-23 18:37:49:mr_type: 356.658 INFO:Trainer:2024-05-23 18:37:49:mrr_type: 0.195 INFO:Trainer:2024-05-23 18:37:49:hits@1_type: 0.146 INFO:Trainer:2024-05-23 18:37:49:hits@3_type: 0.203 INFO:Trainer:2024-05-23 18:37:49:hits@10_type: 0.287 INFO:EarlyStopping:2024-05-23 18:37:49:Validation score improved (0.127000 --> 0.130000). Saving model ... DEBUG:connectionpool:2024-05-23 18:37:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:37:50:[cuda:0] Epoch [2700/10000] | Batchsize: 4096 | loss: 0.089309 | 6.27182 seconds/epoch DEBUG:connectionpool:2024-05-23 18:37:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:38:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:38:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:38:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:38:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:38:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:38:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:38:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:38:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:38:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:38:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:38:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:38:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:39:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:39:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:39:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:39:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:39:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:39:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:39:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:39:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:39:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:39:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:39:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:40:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:40:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:40:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:40:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:40:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:40:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:40:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:40:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:40:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:40:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:40:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:41:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:41:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:41:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:41:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:41:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:41:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:41:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:41:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:41:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:41:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:41:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:41:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:42:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:42:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:42:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:42:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:42:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:42:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:42:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:42:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:42:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:42:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:42:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:43:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:43:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:43:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:43:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:43:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:43:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:43:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:43:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:43:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:43:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:43:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:43:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:44:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:44:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:44:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:44:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:44:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:44:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:44:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:44:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:44:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:44:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:44:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:45:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:45:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:45:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:45:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:45:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:45:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:45:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:45:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:45:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:45:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:45:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:46:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:46:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:46:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:46:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:46:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:46:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:46:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:46:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:46:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:46:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:46:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:46:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:47:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:47:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:47:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:47:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:47:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:47:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:47:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:47:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:47:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:47:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:47:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:48:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:48:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:48:12:[cuda:0] Epoch 2800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 18:48:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:48:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:48:19:mr: 4860.837 INFO:Trainer:2024-05-23 18:48:19:mrr: 0.098 INFO:Trainer:2024-05-23 18:48:19:hits@1: 0.081 INFO:Trainer:2024-05-23 18:48:19:hits@3: 0.103 INFO:Trainer:2024-05-23 18:48:19:hits@10: 0.132 INFO:Trainer:2024-05-23 18:48:19:mr_type: 356.857 INFO:Trainer:2024-05-23 18:48:19:mrr_type: 0.197 INFO:Trainer:2024-05-23 18:48:19:hits@1_type: 0.147 INFO:Trainer:2024-05-23 18:48:19:hits@3_type: 0.204 INFO:Trainer:2024-05-23 18:48:19:hits@10_type: 0.29 INFO:EarlyStopping:2024-05-23 18:48:19:Validation score improved (0.130000 --> 0.132000). Saving model ... INFO:Trainer:2024-05-23 18:48:19:[cuda:0] Epoch [2800/10000] | Batchsize: 4096 | loss: 0.089132 | 6.27246 seconds/epoch DEBUG:connectionpool:2024-05-23 18:48:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:48:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:48:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:48:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:48:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:48:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:48:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:49:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:49:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:49:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:49:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:49:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:49:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:49:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:49:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:49:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:49:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:49:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:49:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:50:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:50:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:50:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:50:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:50:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:50:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:50:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:50:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:50:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:50:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:50:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:51:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:51:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:51:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:51:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:51:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:51:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:51:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:51:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:51:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:51:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:51:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:52:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:52:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:52:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:52:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:52:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:52:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:52:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:52:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:52:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:52:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:52:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:52:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:53:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:53:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:53:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:53:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:53:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:53:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:53:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:53:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:53:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:53:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:53:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:54:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:54:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:54:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:54:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:54:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:54:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:54:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:54:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:54:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:54:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:54:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:55:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:55:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:55:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:55:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:55:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:55:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:55:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:55:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:55:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:55:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:55:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:55:58:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-23 18:55:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:56:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:56:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:56:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:56:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:56:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:56:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:56:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:56:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:56:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:56:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:56:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:57:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:57:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:57:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:57:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:57:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:57:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:57:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:57:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:57:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:57:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:57:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:58:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:58:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:58:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:58:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:58:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:58:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:58:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:58:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:58:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:58:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:58:52:[cuda:0] Epoch 2900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 18:58:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:58:59:mr: 4818.431 INFO:Trainer:2024-05-23 18:58:59:mrr: 0.1 INFO:Trainer:2024-05-23 18:58:59:hits@1: 0.082 INFO:Trainer:2024-05-23 18:58:59:hits@3: 0.105 INFO:Trainer:2024-05-23 18:58:59:hits@10: 0.135 INFO:Trainer:2024-05-23 18:58:59:mr_type: 355.52 INFO:Trainer:2024-05-23 18:58:59:mrr_type: 0.197 INFO:Trainer:2024-05-23 18:58:59:hits@1_type: 0.147 INFO:Trainer:2024-05-23 18:58:59:hits@3_type: 0.205 INFO:Trainer:2024-05-23 18:58:59:hits@10_type: 0.291 INFO:EarlyStopping:2024-05-23 18:58:59:Validation score improved (0.132000 --> 0.135000). Saving model ... DEBUG:connectionpool:2024-05-23 18:58:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 18:58:59:[cuda:0] Epoch [2900/10000] | Batchsize: 4096 | loss: 0.088786 | 6.27665 seconds/epoch DEBUG:connectionpool:2024-05-23 18:59:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:59:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:59:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:59:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:59:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:59:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:59:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:59:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:59:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:59:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 18:59:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:00:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:00:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:00:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:00:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:00:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:00:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:00:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:00:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:00:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:00:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:00:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:01:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:01:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:01:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:01:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:01:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:01:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:01:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:01:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:01:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:01:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:01:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:01:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:02:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:02:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:02:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:02:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:02:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:02:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:02:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:02:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:02:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:02:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:02:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:03:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:03:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:03:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:03:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:03:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:03:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:03:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:03:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:03:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:03:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:03:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:04:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:04:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:04:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:04:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:04:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:04:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:04:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:04:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:04:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:04:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:04:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:04:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:05:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:05:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:05:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:05:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:05:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:05:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:05:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:05:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:05:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:05:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:05:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:06:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:06:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:06:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:06:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:06:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:06:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:06:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:06:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:06:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:06:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:06:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:07:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:07:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:07:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:07:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:07:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:07:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:07:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:07:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:07:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:07:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:07:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:07:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:08:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:08:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:08:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:08:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:08:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:08:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:08:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:08:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:08:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:08:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:08:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:09:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:09:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:09:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:09:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:09:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:09:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:09:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:09:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 19:09:41:[cuda:0] Epoch 3000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 19:09:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 19:09:48:mr: 4775.487 INFO:Trainer:2024-05-23 19:09:48:mrr: 0.102 INFO:Trainer:2024-05-23 19:09:48:hits@1: 0.084 INFO:Trainer:2024-05-23 19:09:48:hits@3: 0.107 INFO:Trainer:2024-05-23 19:09:48:hits@10: 0.138 INFO:Trainer:2024-05-23 19:09:48:mr_type: 354.562 INFO:Trainer:2024-05-23 19:09:48:mrr_type: 0.199 INFO:Trainer:2024-05-23 19:09:48:hits@1_type: 0.148 INFO:Trainer:2024-05-23 19:09:48:hits@3_type: 0.207 INFO:Trainer:2024-05-23 19:09:48:hits@10_type: 0.293 INFO:EarlyStopping:2024-05-23 19:09:48:Validation score improved (0.135000 --> 0.138000). Saving model ... INFO:Trainer:2024-05-23 19:09:48:[cuda:0] Epoch [3000/10000] | Batchsize: 4096 | loss: 0.088680 | 6.28389 seconds/epoch DEBUG:connectionpool:2024-05-23 19:09:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:09:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:10:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:10:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:10:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:10:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:10:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:10:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:10:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:10:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:10:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:10:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:10:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:10:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:11:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:11:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:11:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:11:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:11:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:11:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:11:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:11:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:11:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:11:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:11:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:12:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:12:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:12:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:12:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:12:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:12:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:12:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:12:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:12:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:12:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:12:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:12:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:13:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:13:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:13:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:13:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:13:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:13:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:13:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:13:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:13:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:13:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:13:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:14:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:14:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:14:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:14:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:14:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:14:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:14:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:14:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:14:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:14:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:14:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:15:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:15:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:15:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:15:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:15:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:15:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:15:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:15:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:15:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:15:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:15:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:15:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:16:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:16:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:16:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:16:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:16:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:16:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:16:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:16:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:16:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:16:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:16:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:17:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:17:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:17:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:17:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:17:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:17:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:17:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:17:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:17:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:17:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:17:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:18:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:18:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:18:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:18:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:18:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:18:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:18:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:18:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:18:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:18:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:18:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:18:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:19:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:19:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:19:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:19:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:19:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:19:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:19:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:19:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:19:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:19:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:19:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:20:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:20:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:20:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:20:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:20:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:20:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 19:20:28:[cuda:0] Epoch 3100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 19:20:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 19:20:35:mr: 4743.8 INFO:Trainer:2024-05-23 19:20:35:mrr: 0.104 INFO:Trainer:2024-05-23 19:20:35:hits@1: 0.085 INFO:Trainer:2024-05-23 19:20:35:hits@3: 0.108 INFO:Trainer:2024-05-23 19:20:35:hits@10: 0.142 INFO:Trainer:2024-05-23 19:20:35:mr_type: 354.185 INFO:Trainer:2024-05-23 19:20:35:mrr_type: 0.199 INFO:Trainer:2024-05-23 19:20:35:hits@1_type: 0.149 INFO:Trainer:2024-05-23 19:20:35:hits@3_type: 0.208 INFO:Trainer:2024-05-23 19:20:35:hits@10_type: 0.293 INFO:EarlyStopping:2024-05-23 19:20:35:Validation score improved (0.138000 --> 0.142000). Saving model ... INFO:Trainer:2024-05-23 19:20:35:[cuda:0] Epoch [3100/10000] | Batchsize: 4096 | loss: 0.088474 | 6.28995 seconds/epoch DEBUG:connectionpool:2024-05-23 19:20:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:20:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:20:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:20:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:21:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:21:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:21:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:21:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:21:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:21:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:21:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:21:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:21:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:21:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:21:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:21:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:22:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:22:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:22:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:22:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:22:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:22:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:22:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:22:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:22:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:22:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:22:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:23:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:23:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:23:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:23:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:23:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:23:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:23:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:23:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:23:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:23:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:23:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:24:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:24:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:24:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:24:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:24:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:24:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:24:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:24:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:24:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:24:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:24:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:24:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:25:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:25:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:25:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:25:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:25:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:25:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:25:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:25:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:25:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:25:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:25:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:26:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:26:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:26:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:26:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:26:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:26:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:26:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:26:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:26:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:26:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:26:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:27:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:27:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:27:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:27:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:27:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:27:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:27:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:27:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:27:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:27:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:27:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:27:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:28:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:28:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:28:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:28:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:28:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:28:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:28:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:28:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:28:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:28:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:28:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:29:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:29:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:29:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:29:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:29:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:29:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:29:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:29:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:29:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:29:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:29:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:30:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:30:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:30:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:30:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:30:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:30:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:30:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:30:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:30:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:30:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:30:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:30:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:31:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:31:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:31:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 19:31:13:[cuda:0] Epoch 3200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 19:31:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 19:31:20:mr: 4708.929 INFO:Trainer:2024-05-23 19:31:20:mrr: 0.105 INFO:Trainer:2024-05-23 19:31:20:hits@1: 0.086 INFO:Trainer:2024-05-23 19:31:20:hits@3: 0.11 INFO:Trainer:2024-05-23 19:31:20:hits@10: 0.145 INFO:Trainer:2024-05-23 19:31:20:mr_type: 353.714 INFO:Trainer:2024-05-23 19:31:20:mrr_type: 0.2 INFO:Trainer:2024-05-23 19:31:20:hits@1_type: 0.15 INFO:Trainer:2024-05-23 19:31:20:hits@3_type: 0.209 INFO:Trainer:2024-05-23 19:31:20:hits@10_type: 0.295 INFO:EarlyStopping:2024-05-23 19:31:20:Validation score improved (0.142000 --> 0.145000). Saving model ... INFO:Trainer:2024-05-23 19:31:20:[cuda:0] Epoch [3200/10000] | Batchsize: 4096 | loss: 0.088210 | 6.29497 seconds/epoch DEBUG:connectionpool:2024-05-23 19:31:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:31:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:31:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:31:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:31:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:31:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:31:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:32:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:32:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:32:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:32:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:32:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:32:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:32:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:32:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:32:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:32:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:32:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:32:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:33:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:33:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:33:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:33:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:33:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:33:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:33:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:33:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:33:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:33:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:33:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:34:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:34:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:34:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:34:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:34:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:34:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:34:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:34:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:34:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:34:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:34:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:35:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:35:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:35:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:35:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:35:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:35:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:35:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:35:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:35:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:35:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:35:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:35:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:36:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:36:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:36:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:36:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:36:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:36:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:36:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:36:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:36:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:36:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:36:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:37:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:37:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:37:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:37:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:37:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:37:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:37:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:37:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:37:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:37:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:37:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:38:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:38:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:38:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:38:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:38:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:38:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:38:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:38:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:38:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:38:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:38:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:38:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:39:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:39:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:39:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:39:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:39:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:39:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:39:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:39:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:39:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:39:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:39:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:40:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:40:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:40:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:40:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:40:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:40:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:40:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:40:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:40:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:40:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:40:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:41:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:41:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:41:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:41:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:41:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:41:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:41:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:41:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:41:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:41:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:41:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:41:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 19:42:00:[cuda:0] Epoch 3300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 19:42:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 19:42:07:mr: 4675.307 INFO:Trainer:2024-05-23 19:42:07:mrr: 0.107 INFO:Trainer:2024-05-23 19:42:07:hits@1: 0.088 INFO:Trainer:2024-05-23 19:42:07:hits@3: 0.112 INFO:Trainer:2024-05-23 19:42:07:hits@10: 0.148 INFO:Trainer:2024-05-23 19:42:07:mr_type: 353.174 INFO:Trainer:2024-05-23 19:42:07:mrr_type: 0.201 INFO:Trainer:2024-05-23 19:42:07:hits@1_type: 0.15 INFO:Trainer:2024-05-23 19:42:07:hits@3_type: 0.21 INFO:Trainer:2024-05-23 19:42:07:hits@10_type: 0.297 INFO:EarlyStopping:2024-05-23 19:42:07:Validation score improved (0.145000 --> 0.148000). Saving model ... INFO:Trainer:2024-05-23 19:42:07:[cuda:0] Epoch [3300/10000] | Batchsize: 4096 | loss: 0.088083 | 6.30009 seconds/epoch DEBUG:connectionpool:2024-05-23 19:42:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:42:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:42:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:42:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:42:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:42:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:42:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:42:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:42:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:42:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:43:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:43:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:43:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:43:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:43:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:43:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:43:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:43:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:43:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:43:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:43:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:44:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:44:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:44:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:44:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:44:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:44:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:44:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:44:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:44:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:44:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:44:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:44:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:45:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:45:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:45:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:45:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:45:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:45:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:45:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:45:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:45:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:45:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:45:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:46:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:46:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:46:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:46:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:46:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:46:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:46:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:46:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:46:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:46:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:46:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:47:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:47:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:47:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:47:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:47:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:47:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:47:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:47:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:47:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:47:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:47:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:47:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:48:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:48:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:48:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:48:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:48:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:48:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:48:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:48:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:48:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:48:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:48:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:49:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:49:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:49:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:49:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:49:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:49:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:49:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:49:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:49:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:49:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:49:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:50:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:50:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:50:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:50:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:50:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:50:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:50:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:50:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:50:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:50:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:50:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:50:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:51:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:51:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:51:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:51:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:51:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:51:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:51:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:51:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:51:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:51:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:51:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:52:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:52:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:52:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:52:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:52:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:52:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:52:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:52:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 19:52:42:[cuda:0] Epoch 3400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 19:52:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 19:52:49:mr: 4662.845 INFO:Trainer:2024-05-23 19:52:49:mrr: 0.108 INFO:Trainer:2024-05-23 19:52:49:hits@1: 0.088 INFO:Trainer:2024-05-23 19:52:49:hits@3: 0.113 INFO:Trainer:2024-05-23 19:52:49:hits@10: 0.149 INFO:Trainer:2024-05-23 19:52:49:mr_type: 352.957 INFO:Trainer:2024-05-23 19:52:49:mrr_type: 0.201 INFO:Trainer:2024-05-23 19:52:49:hits@1_type: 0.15 INFO:Trainer:2024-05-23 19:52:49:hits@3_type: 0.21 INFO:Trainer:2024-05-23 19:52:49:hits@10_type: 0.297 INFO:EarlyStopping:2024-05-23 19:52:49:Validation score improved (0.148000 --> 0.149000). Saving model ... INFO:Trainer:2024-05-23 19:52:49:[cuda:0] Epoch [3400/10000] | Batchsize: 4096 | loss: 0.087846 | 6.30367 seconds/epoch DEBUG:connectionpool:2024-05-23 19:52:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:52:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:53:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:53:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:53:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:53:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:53:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:53:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:53:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:53:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:53:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:53:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:53:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:54:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:54:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:54:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:54:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:54:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:54:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:54:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:54:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:54:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:54:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:54:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:55:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:55:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:55:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:55:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:55:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:55:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:55:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:55:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:55:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:55:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:55:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:56:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:56:06:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-23 19:56:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:56:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:56:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:56:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:56:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:56:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:56:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:56:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:56:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:56:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:56:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:57:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:57:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:57:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:57:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:57:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:57:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:57:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:57:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:57:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:57:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:57:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:58:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:58:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:58:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:58:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:58:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:58:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:58:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:58:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:58:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:58:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:58:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:59:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:59:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:59:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:59:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:59:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:59:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:59:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:59:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:59:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:59:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:59:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 19:59:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:00:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:00:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:00:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:00:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:00:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:00:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:00:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:00:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:00:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:00:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:00:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:01:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:01:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:01:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:01:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:01:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:01:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:01:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:01:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:01:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:01:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:01:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:02:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:02:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:02:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:02:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:02:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:02:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:02:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:02:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:02:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:02:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:02:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:02:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:03:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:03:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:03:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:03:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:03:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 20:03:28:[cuda:0] Epoch 3500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 20:03:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 20:03:34:mr: 4659.543 INFO:Trainer:2024-05-23 20:03:34:mrr: 0.108 INFO:Trainer:2024-05-23 20:03:34:hits@1: 0.088 INFO:Trainer:2024-05-23 20:03:34:hits@3: 0.113 INFO:Trainer:2024-05-23 20:03:34:hits@10: 0.149 INFO:Trainer:2024-05-23 20:03:34:mr_type: 352.867 INFO:Trainer:2024-05-23 20:03:34:mrr_type: 0.201 INFO:Trainer:2024-05-23 20:03:34:hits@1_type: 0.15 INFO:Trainer:2024-05-23 20:03:34:hits@3_type: 0.21 INFO:Trainer:2024-05-23 20:03:34:hits@10_type: 0.297 INFO:EarlyStopping:2024-05-23 20:03:34:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-05-23 20:03:34:[cuda:0] Epoch [3500/10000] | Batchsize: 4096 | loss: 0.087986 | 6.30808 seconds/epoch DEBUG:connectionpool:2024-05-23 20:03:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:03:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:03:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:03:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:03:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:04:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:04:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:04:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:04:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:04:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:04:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:04:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:04:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:04:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:04:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:04:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:05:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:05:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:05:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:05:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:05:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:05:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:05:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:05:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:05:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:05:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:05:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:05:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:06:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:06:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:06:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:06:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:06:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:06:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:06:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:06:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:06:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:06:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:06:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:07:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:07:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:07:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:07:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:07:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:07:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:07:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:07:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:07:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:07:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:07:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:08:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:08:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:08:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:08:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:08:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:08:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:08:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:08:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:08:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:08:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:08:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:08:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:09:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:09:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:09:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:09:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:09:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:09:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:09:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:09:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:09:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:09:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:09:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:10:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:10:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:10:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:10:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:10:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:10:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:10:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:10:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:10:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:10:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:10:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:11:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:11:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:11:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:11:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:11:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:11:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:11:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:11:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:11:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:11:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:11:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:12:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:12:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:12:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:12:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:12:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:12:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:12:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:12:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:12:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:12:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:12:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:12:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:13:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:13:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:13:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:13:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:13:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:13:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:13:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:13:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:13:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:13:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:13:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:14:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:14:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:14:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 20:14:13:[cuda:0] Epoch 3600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 20:14:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 20:14:19:mr: 4656.357 INFO:Trainer:2024-05-23 20:14:19:mrr: 0.108 INFO:Trainer:2024-05-23 20:14:19:hits@1: 0.088 INFO:Trainer:2024-05-23 20:14:19:hits@3: 0.113 INFO:Trainer:2024-05-23 20:14:19:hits@10: 0.149 INFO:Trainer:2024-05-23 20:14:19:mr_type: 352.814 INFO:Trainer:2024-05-23 20:14:19:mrr_type: 0.201 INFO:Trainer:2024-05-23 20:14:19:hits@1_type: 0.15 INFO:Trainer:2024-05-23 20:14:19:hits@3_type: 0.21 INFO:Trainer:2024-05-23 20:14:19:hits@10_type: 0.298 INFO:EarlyStopping:2024-05-23 20:14:19:EarlyStopping counter: 2 / 2 INFO:Trainer:2024-05-23 20:14:19:[cuda:0] Send an early stopping signal INFO:Trainer:2024-05-23 20:14:19:[cuda:0] The model training is completed, taking a total of 22722.98204 seconds. INFO:Trainer:2024-05-23 20:14:19:[cuda:0] Model saved at ./model.pth. INFO:Trainer:2024-05-23 20:14:19:[cuda:0] The model starts evaluating in the test set. DEBUG:connectionpool:2024-05-23 20:14:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 20:14:27:mr: 4677.491 INFO:Trainer:2024-05-23 20:14:27:mrr: 0.106 INFO:Trainer:2024-05-23 20:14:27:hits@1: 0.086 INFO:Trainer:2024-05-23 20:14:27:hits@3: 0.111 INFO:Trainer:2024-05-23 20:14:27:hits@10: 0.148 INFO:Trainer:2024-05-23 20:14:27:mr_type: 360.797 INFO:Trainer:2024-05-23 20:14:27:mrr_type: 0.198 INFO:Trainer:2024-05-23 20:14:27:hits@1_type: 0.146 INFO:Trainer:2024-05-23 20:14:27:hits@3_type: 0.207 INFO:Trainer:2024-05-23 20:14:27:hits@10_type: 0.294 DEBUG:connectionpool:2024-05-23 20:14:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: - 0.014 MB of 0.014 MB uploaded wandb: \ 0.016 MB of 0.492 MB uploaded wandb: | 0.456 MB of 0.492 MB uploaded wandb: / 0.492 MB of 0.492 MB uploaded wandb: - 0.492 MB of 0.492 MB uploaded wandb: \ 0.492 MB of 0.492 MB uploaded wandb: DEBUG:connectionpool:2024-05-23 20:14:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: wandb: Run history: wandb: duration โ– wandb: test/hits@1 โ– wandb: test/hits@10 โ– wandb: test/hits@10_type โ– wandb: test/hits@1_type โ– wandb: test/hits@3 โ– wandb: test/hits@3_type โ– wandb: test/mr โ– wandb: test/mr_type โ– wandb: test/mrr โ– wandb: test/mrr_type โ– wandb: train/epoch โ–โ–โ–โ–‚โ–‚โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆ wandb: train/train_loss โ–ˆโ–‡โ–†โ–†โ–…โ–…โ–„โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ– wandb: val/epoch โ–โ–โ–โ–‚โ–‚โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆ wandb: val/hits@1 โ–โ–โ–‚โ–ƒโ–„โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@10 โ–โ–‚โ–‚โ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@10_type โ–โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@1_type โ–โ–‚โ–ƒโ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@3 โ–โ–โ–‚โ–ƒโ–„โ–„โ–…โ–…โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@3_type โ–โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/mr โ–ˆโ–‡โ–‡โ–‡โ–†โ–†โ–†โ–…โ–…โ–…โ–…โ–„โ–„โ–„โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ– wandb: val/mr_type โ–ˆโ–ˆโ–‡โ–‡โ–‡โ–‡โ–†โ–†โ–†โ–†โ–†โ–…โ–…โ–…โ–…โ–…โ–„โ–„โ–„โ–„โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ– wandb: val/mrr โ–โ–โ–‚โ–ƒโ–„โ–„โ–…โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/mrr_type โ–โ–‚โ–‚โ–ƒโ–ƒโ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: wandb: Run summary: wandb: duration 22722.98204 wandb: test/hits@1 0.086 wandb: test/hits@10 0.148 wandb: test/hits@10_type 0.294 wandb: test/hits@1_type 0.146 wandb: test/hits@3 0.111 wandb: test/hits@3_type 0.207 wandb: test/mr 4677.491 wandb: test/mr_type 360.797 wandb: test/mrr 0.106 wandb: test/mrr_type 0.198 wandb: train/epoch 3500 wandb: train/train_loss 0.08799 wandb: val/epoch 3599 wandb: val/hits@1 0.088 wandb: val/hits@10 0.149 wandb: val/hits@10_type 0.298 wandb: val/hits@1_type 0.15 wandb: val/hits@3 0.113 wandb: val/hits@3_type 0.21 wandb: val/mr 4656.357 wandb: val/mr_type 352.814 wandb: val/mrr 0.108 wandb: val/mrr_type 0.201 wandb: wandb: ๐Ÿš€ View run confused-sweep-1 at: https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/343tmdcx wandb: โญ๏ธ View project at: https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) wandb: Find logs at: ./wandb/run-20240523_135517-343tmdcx/logs DEBUG:connectionpool:2024-05-23 20:14:34:Starting new HTTPS connection (1): o151352.ingest.sentry.io:443 DEBUG:connectionpool:2024-05-23 20:14:35:https://o151352.ingest.sentry.io:443 "POST /api/4504800232407040/envelope/ HTTP/1.1" 200 0 DEBUG:connectionpool:2024-05-23 20:14:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: Agent Starting Run: 6bgcsi0p with config: wandb: batch_size: 1024 wandb: dataloader: KGEDataLoader wandb: decoder_model: ConvE wandb: delta: 0.0001 wandb: device: cuda:0 wandb: dim: 100 wandb: ent_file: entity2id.txt wandb: epochs: 10000 wandb: fet_drop: 0.2 wandb: hid_drop: 0.3 wandb: in_path: ../../benchmarks/FB15K237/ wandb: log_interval: 100 wandb: loss: CompGCNLoss wandb: lr: 0.003693217884549537 wandb: margin: 40 wandb: metric: hits@10 wandb: model: CompGCN wandb: neg_ent: 1 wandb: num_workers: 16 wandb: opn: mult wandb: opt_method: adagrad wandb: patience: 2 wandb: prediction: tail wandb: rel_file: relation2id.txt wandb: save_path: ./model.pth wandb: smoothing: 0.1 wandb: strategy: CompGCNSampling wandb: test_batch_size: 256 wandb: test_file: test2id.txt wandb: test_sampler: CompGCNTestSampler wandb: tester: Tester wandb: train_file: train2id.txt wandb: train_sampler: CompGCNSampler wandb: trainer: Trainer wandb: type_constrain: True wandb: use_early_stopping: True wandb: use_gpu: True wandb: use_tqdm: False wandb: valid_file: valid2id.txt wandb: valid_interval: 100 DEBUG:cmd:2024-05-23 20:14:39:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:cmd:2024-05-23 20:14:39:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-05-23 20:14:40:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-05-23 20:14:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:14:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:cmd:2024-05-23 20:14:42:Popen(['git', 'cat-file', '--batch-check'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=, shell=False, universal_newlines=False) wandb: wandb version 0.17.0 is available! To upgrade, please run: wandb: $ pip install wandb --upgrade wandb: Tracking run with wandb version 0.16.6 wandb: Run data is saved locally in /home/luyanfeng/my_code/github/pybind11-OpenKE/examples/CompGCN/wandb/run-20240523_201441-6bgcsi0p wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run worldly-sweep-2 wandb: โญ๏ธ View project at https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: ๐Ÿงน View sweep at https://wandb.ai/3555028709/pybind11-ke-sweeps/sweeps/fnx5v46t wandb: ๐Ÿš€ View run at https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/6bgcsi0p DEBUG:connectionpool:2024-05-23 20:14:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 20:14:49:[cuda:0] Initialization completed, start model training. DEBUG:connectionpool:2024-05-23 20:14:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:14:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:15:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:15:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:15:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:15:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:15:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:15:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:15:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:15:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:15:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:15:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:15:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:15:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:16:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:16:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:16:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:16:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:16:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:16:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:16:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:16:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:16:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:16:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:16:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:17:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:17:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:17:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:17:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:17:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:17:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:17:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:17:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:17:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:17:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:17:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:17:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:18:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:18:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:18:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:18:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:18:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:18:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:18:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:18:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:18:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:18:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:18:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:19:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:19:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:19:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:19:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:19:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:19:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:19:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:19:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:19:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:19:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:19:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:20:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:20:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:20:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:20:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:20:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:20:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:20:33:Starting new HTTPS connection (2): api.wandb.ai:443 DEBUG:connectionpool:2024-05-23 20:20:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:20:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:20:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:20:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:20:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:21:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:21:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:21:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:21:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:21:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:21:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:21:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:21:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:21:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:21:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:21:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:22:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:22:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:22:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:22:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:22:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:22:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:22:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:22:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:22:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:22:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:22:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:22:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:23:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:23:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:23:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:23:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:23:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:23:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:23:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:23:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:23:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:23:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:23:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:24:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:24:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:24:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:24:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:24:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:24:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:24:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:24:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:24:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:24:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:24:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:24:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:25:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:25:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:25:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:25:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:25:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:25:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:25:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:25:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:25:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:25:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:25:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:26:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:26:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:26:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:26:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:26:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:26:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:26:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:26:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:26:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:26:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:26:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:27:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 20:27:03:[cuda:0] Epoch 100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 20:27:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 20:27:09:mr: 2965.586 INFO:Trainer:2024-05-23 20:27:09:mrr: 0.124 INFO:Trainer:2024-05-23 20:27:09:hits@1: 0.096 INFO:Trainer:2024-05-23 20:27:09:hits@3: 0.13 INFO:Trainer:2024-05-23 20:27:09:hits@10: 0.179 INFO:Trainer:2024-05-23 20:27:09:mr_type: 313.878 INFO:Trainer:2024-05-23 20:27:09:mrr_type: 0.23 INFO:Trainer:2024-05-23 20:27:09:hits@1_type: 0.171 INFO:Trainer:2024-05-23 20:27:09:hits@3_type: 0.244 INFO:Trainer:2024-05-23 20:27:09:hits@10_type: 0.348 INFO:EarlyStopping:2024-05-23 20:27:09:Validation score improved (-inf --> 0.179000). Saving model ... INFO:Trainer:2024-05-23 20:27:09:[cuda:0] Epoch [ 100/10000] | Batchsize: 1024 | loss: 0.348030 | 7.34123 seconds/epoch DEBUG:connectionpool:2024-05-23 20:27:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:27:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:27:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:27:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:27:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:27:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:27:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:27:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:27:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:27:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:28:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:28:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:28:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:28:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:28:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:28:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:28:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:28:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:28:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:28:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:28:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:29:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:29:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:29:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:29:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:29:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:29:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:29:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:29:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:29:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:29:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:29:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:29:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:30:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:30:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:30:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:30:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:30:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:30:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:30:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:30:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:30:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:30:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:30:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:31:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:31:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:31:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:31:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:31:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:31:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:31:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:31:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:31:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:31:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:31:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:32:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:32:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:32:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:32:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:32:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:32:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:32:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:32:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:32:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:32:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:32:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:32:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:33:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:33:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:33:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:33:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:33:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:33:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:33:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:33:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:33:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:33:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:33:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:34:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:34:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:34:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:34:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:34:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:34:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:34:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:34:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:34:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:34:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:34:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:35:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:35:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:35:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:35:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:35:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:35:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:35:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:35:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:35:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:35:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:35:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:35:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:36:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:36:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:36:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:36:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:36:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:36:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:36:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:36:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:36:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:36:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:36:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:37:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:37:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:37:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:37:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:37:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:37:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:37:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:37:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:37:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:37:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:37:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:38:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:38:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:38:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:38:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:38:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:38:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:38:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:38:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:38:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:38:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:38:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:39:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:39:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:39:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:39:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:39:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 20:39:30:[cuda:0] Epoch 200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 20:39:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:39:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 20:39:37:mr: 1640.281 INFO:Trainer:2024-05-23 20:39:37:mrr: 0.177 INFO:Trainer:2024-05-23 20:39:37:hits@1: 0.13 INFO:Trainer:2024-05-23 20:39:37:hits@3: 0.191 INFO:Trainer:2024-05-23 20:39:37:hits@10: 0.27 INFO:Trainer:2024-05-23 20:39:37:mr_type: 304.198 INFO:Trainer:2024-05-23 20:39:37:mrr_type: 0.241 INFO:Trainer:2024-05-23 20:39:37:hits@1_type: 0.179 INFO:Trainer:2024-05-23 20:39:37:hits@3_type: 0.257 INFO:Trainer:2024-05-23 20:39:37:hits@10_type: 0.361 INFO:EarlyStopping:2024-05-23 20:39:37:Validation score improved (0.179000 --> 0.270000). Saving model ... INFO:Trainer:2024-05-23 20:39:37:[cuda:0] Epoch [ 200/10000] | Batchsize: 1024 | loss: 0.314703 | 7.40598 seconds/epoch DEBUG:connectionpool:2024-05-23 20:39:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:39:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:39:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:39:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:40:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:40:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:40:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:40:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:40:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:40:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:40:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:40:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:40:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:40:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:40:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:41:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:41:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:41:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:41:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:41:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:41:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:41:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:41:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:41:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:41:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:41:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:42:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:42:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:42:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:42:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:42:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:42:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:42:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:42:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:42:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:42:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:42:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:42:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:43:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:43:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:43:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:43:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:43:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:43:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:43:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:43:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:43:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:43:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:43:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:44:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:44:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:44:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:44:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:44:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:44:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:44:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:44:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:44:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:44:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:44:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:44:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:45:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:45:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:45:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:45:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:45:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:45:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:45:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:45:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:45:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:45:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:45:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:46:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:46:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:46:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:46:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:46:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:46:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:46:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:46:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:46:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:46:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:46:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:47:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:47:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:47:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:47:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:47:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:47:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:47:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:47:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:47:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:47:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:47:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:47:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:48:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:48:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:48:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:48:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:48:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:48:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:48:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:48:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:48:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:48:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:48:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:49:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:49:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:49:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:49:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:49:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:49:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:49:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:49:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:49:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:49:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:49:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:49:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:50:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:50:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:50:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:50:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:50:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:50:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:50:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:50:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:50:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:50:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:50:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:51:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:51:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:51:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:51:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:51:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:51:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:51:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:51:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:51:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:51:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:51:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:51:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:52:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 20:52:10:[cuda:0] Epoch 300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 20:52:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:52:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 20:52:16:mr: 945.962 INFO:Trainer:2024-05-23 20:52:16:mrr: 0.21 INFO:Trainer:2024-05-23 20:52:16:hits@1: 0.154 INFO:Trainer:2024-05-23 20:52:16:hits@3: 0.228 INFO:Trainer:2024-05-23 20:52:16:hits@10: 0.32 INFO:Trainer:2024-05-23 20:52:16:mr_type: 275.809 INFO:Trainer:2024-05-23 20:52:16:mrr_type: 0.252 INFO:Trainer:2024-05-23 20:52:16:hits@1_type: 0.187 INFO:Trainer:2024-05-23 20:52:16:hits@3_type: 0.27 INFO:Trainer:2024-05-23 20:52:16:hits@10_type: 0.378 INFO:EarlyStopping:2024-05-23 20:52:16:Validation score improved (0.270000 --> 0.320000). Saving model ... INFO:Trainer:2024-05-23 20:52:17:[cuda:0] Epoch [ 300/10000] | Batchsize: 1024 | loss: 0.294053 | 7.47006 seconds/epoch DEBUG:connectionpool:2024-05-23 20:52:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:52:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:52:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:52:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:52:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:52:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:52:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:52:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:53:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:53:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:53:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:53:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:53:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:53:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:53:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:53:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:53:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:53:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:53:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:54:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:54:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:54:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:54:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:54:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:54:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:54:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:54:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:54:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:54:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:54:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:54:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:55:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:55:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:55:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:55:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:55:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:55:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:55:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:55:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:55:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:55:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:55:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:56:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:56:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:56:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:56:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:56:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:56:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:56:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:56:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:56:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:56:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:56:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:57:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:57:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:57:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:57:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:57:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:57:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:57:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:57:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:57:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:57:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:57:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:57:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:58:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:58:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:58:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:58:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:58:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:58:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:58:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:58:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:58:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:58:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:58:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:59:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:59:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:59:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:59:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:59:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:59:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:59:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:59:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:59:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:59:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 20:59:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:00:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:00:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:00:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:00:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:00:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:00:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:00:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:00:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:00:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:00:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:00:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:00:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:01:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:01:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:01:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:01:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:01:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:01:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:01:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:01:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:01:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:01:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:01:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:02:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:02:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:02:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:02:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:02:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:02:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:02:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:02:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:02:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:02:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:02:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:02:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:03:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:03:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:03:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:03:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:03:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:03:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:03:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:03:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:03:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:03:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:03:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:04:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:04:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:04:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:04:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:04:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:04:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:04:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:04:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:04:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 21:04:47:[cuda:0] Epoch 400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 21:04:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:04:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 21:04:56:mr: 653.9 INFO:Trainer:2024-05-23 21:04:56:mrr: 0.23 INFO:Trainer:2024-05-23 21:04:56:hits@1: 0.169 INFO:Trainer:2024-05-23 21:04:56:hits@3: 0.249 INFO:Trainer:2024-05-23 21:04:56:hits@10: 0.35 INFO:Trainer:2024-05-23 21:04:56:mr_type: 250.195 INFO:Trainer:2024-05-23 21:04:56:mrr_type: 0.263 INFO:Trainer:2024-05-23 21:04:56:hits@1_type: 0.195 INFO:Trainer:2024-05-23 21:04:56:hits@3_type: 0.282 INFO:Trainer:2024-05-23 21:04:56:hits@10_type: 0.395 INFO:EarlyStopping:2024-05-23 21:04:56:Validation score improved (0.320000 --> 0.350000). Saving model ... INFO:Trainer:2024-05-23 21:04:56:[cuda:0] Epoch [ 400/10000] | Batchsize: 1024 | loss: 0.281390 | 7.49576 seconds/epoch DEBUG:connectionpool:2024-05-23 21:05:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:05:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:05:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:05:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:05:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:05:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:05:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:05:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:05:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:05:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:05:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:05:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:06:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:06:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:06:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:06:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:06:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:06:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:06:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:06:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:06:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:06:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:06:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:07:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:07:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:07:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:07:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:07:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:07:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:07:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:07:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:07:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:07:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:07:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:08:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:08:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:08:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:08:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:08:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:08:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:08:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:08:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:08:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:08:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:08:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:08:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:09:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:09:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:09:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:09:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:09:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:09:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:09:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:09:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:09:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:09:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:09:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:10:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:10:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:10:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:10:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:10:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:10:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:10:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:10:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:10:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:10:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:10:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:11:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:11:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:11:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:11:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:11:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:11:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:11:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:11:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:11:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:11:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:11:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:11:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:12:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:12:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:12:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:12:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:12:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:12:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:12:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:12:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:12:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:12:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:12:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:13:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:13:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:13:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:13:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:13:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:13:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:13:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:13:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:13:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:13:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:13:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:14:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:14:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:14:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:14:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:14:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:14:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:14:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:14:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:14:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:14:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:14:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:14:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:15:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:15:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:15:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:15:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:15:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:15:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:15:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:15:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:15:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:15:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:15:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:16:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:16:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:16:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:16:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:16:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:16:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:16:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:16:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:16:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:16:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:16:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:16:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:17:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:17:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:17:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:17:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:17:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 21:17:27:[cuda:0] Epoch 500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 21:17:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 21:17:33:mr: 517.413 INFO:Trainer:2024-05-23 21:17:33:mrr: 0.241 INFO:Trainer:2024-05-23 21:17:33:hits@1: 0.176 INFO:Trainer:2024-05-23 21:17:33:hits@3: 0.263 INFO:Trainer:2024-05-23 21:17:33:hits@10: 0.368 INFO:Trainer:2024-05-23 21:17:33:mr_type: 235.253 INFO:Trainer:2024-05-23 21:17:33:mrr_type: 0.27 INFO:Trainer:2024-05-23 21:17:33:hits@1_type: 0.2 INFO:Trainer:2024-05-23 21:17:33:hits@3_type: 0.291 INFO:Trainer:2024-05-23 21:17:33:hits@10_type: 0.404 INFO:EarlyStopping:2024-05-23 21:17:33:Validation score improved (0.350000 --> 0.368000). Saving model ... INFO:Trainer:2024-05-23 21:17:33:[cuda:0] Epoch [ 500/10000] | Batchsize: 1024 | loss: 0.271864 | 7.51535 seconds/epoch DEBUG:connectionpool:2024-05-23 21:17:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:17:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:17:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:17:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:17:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:18:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:18:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:18:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:18:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:18:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:18:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:18:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:18:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:18:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:18:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:18:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:19:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:19:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:19:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:19:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:19:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:19:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:19:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:19:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:19:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:19:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:19:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:19:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:20:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:20:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:20:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:20:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:20:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:20:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:20:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:20:40:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-23 21:20:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:20:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:20:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:20:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:21:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:21:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:21:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:21:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:21:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:21:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:21:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:21:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:21:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:21:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:21:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:22:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:22:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:22:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:22:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:22:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:22:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:22:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:22:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:22:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:22:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:22:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:22:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:23:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:23:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:23:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:23:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:23:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:23:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:23:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:23:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:23:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:23:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:23:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:24:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:24:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:24:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:24:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:24:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:24:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:24:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:24:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:24:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:24:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:24:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:24:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:25:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:25:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:25:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:25:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:25:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:25:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:25:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:25:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:25:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:25:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:25:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:26:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:26:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:26:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:26:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:26:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:26:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:26:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:26:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:26:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:26:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:26:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:27:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:27:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:27:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:27:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:27:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:27:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:27:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:27:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:27:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:27:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:27:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:27:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:28:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:28:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:28:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:28:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:28:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:28:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:28:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:28:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:28:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:28:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:28:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:29:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:29:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:29:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:29:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:29:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:29:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:29:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:29:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:29:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:29:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:29:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:29:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 21:30:00:[cuda:0] Epoch 600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 21:30:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 21:30:06:mr: 450.332 INFO:Trainer:2024-05-23 21:30:06:mrr: 0.251 INFO:Trainer:2024-05-23 21:30:06:hits@1: 0.183 INFO:Trainer:2024-05-23 21:30:06:hits@3: 0.273 INFO:Trainer:2024-05-23 21:30:06:hits@10: 0.382 INFO:Trainer:2024-05-23 21:30:06:mr_type: 223.649 INFO:Trainer:2024-05-23 21:30:06:mrr_type: 0.276 INFO:Trainer:2024-05-23 21:30:06:hits@1_type: 0.205 INFO:Trainer:2024-05-23 21:30:06:hits@3_type: 0.298 INFO:Trainer:2024-05-23 21:30:06:hits@10_type: 0.414 INFO:EarlyStopping:2024-05-23 21:30:06:Validation score improved (0.368000 --> 0.382000). Saving model ... INFO:Trainer:2024-05-23 21:30:06:[cuda:0] Epoch [ 600/10000] | Batchsize: 1024 | loss: 0.265410 | 7.51795 seconds/epoch DEBUG:connectionpool:2024-05-23 21:30:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:30:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:30:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:30:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:30:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:30:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:30:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:30:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:30:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:30:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:31:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:31:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:31:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:31:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:31:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:31:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:31:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:31:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:31:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:31:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:31:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:32:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:32:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:32:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:32:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:32:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:32:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:32:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:32:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:32:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:32:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:32:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:32:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:33:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:33:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:33:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:33:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:33:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:33:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:33:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:33:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:33:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:33:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:33:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:34:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:34:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:34:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:34:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:34:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:34:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:34:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:34:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:34:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:34:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:34:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:34:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:35:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:35:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:35:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:35:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:35:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:35:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:35:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:35:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:35:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:35:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:35:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:36:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:36:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:36:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:36:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:36:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:36:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:36:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:36:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:36:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:36:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:36:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:37:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:37:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:37:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:37:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:37:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:37:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:37:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:37:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:37:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:37:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:37:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:37:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:38:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:38:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:38:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:38:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:38:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:38:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:38:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:38:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:38:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:38:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:38:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:39:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:39:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:39:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:39:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:39:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:39:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:39:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:39:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:39:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:39:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:39:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:40:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:40:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:40:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:40:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:40:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:40:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:40:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:40:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:40:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:40:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:40:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:40:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:41:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:41:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:41:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:41:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:41:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:41:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:41:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:41:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:41:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:41:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:41:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:42:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:42:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:42:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:42:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:42:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:42:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 21:42:30:[cuda:0] Epoch 700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 21:42:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 21:42:36:mr: 412.096 INFO:Trainer:2024-05-23 21:42:36:mrr: 0.257 INFO:Trainer:2024-05-23 21:42:36:hits@1: 0.188 INFO:Trainer:2024-05-23 21:42:36:hits@3: 0.279 INFO:Trainer:2024-05-23 21:42:36:hits@10: 0.391 INFO:Trainer:2024-05-23 21:42:36:mr_type: 212.686 INFO:Trainer:2024-05-23 21:42:36:mrr_type: 0.281 INFO:Trainer:2024-05-23 21:42:36:hits@1_type: 0.209 INFO:Trainer:2024-05-23 21:42:36:hits@3_type: 0.303 INFO:Trainer:2024-05-23 21:42:36:hits@10_type: 0.422 INFO:EarlyStopping:2024-05-23 21:42:36:Validation score improved (0.382000 --> 0.391000). Saving model ... INFO:Trainer:2024-05-23 21:42:36:[cuda:0] Epoch [ 700/10000] | Batchsize: 1024 | loss: 0.260270 | 7.51648 seconds/epoch DEBUG:connectionpool:2024-05-23 21:42:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:42:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:42:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:42:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:43:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:43:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:43:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:43:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:43:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:43:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:43:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:43:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:43:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:43:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:43:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:43:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:44:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:44:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:44:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:44:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:44:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:44:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:44:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:44:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:44:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:44:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:44:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:45:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:45:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:45:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:45:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:45:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:45:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:45:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:45:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:45:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:45:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:45:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:45:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:46:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:46:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:46:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:46:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:46:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:46:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:46:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:46:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:46:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:46:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:46:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:47:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:47:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:47:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:47:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:47:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:47:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:47:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:47:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:47:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:47:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:47:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:48:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:48:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:48:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:48:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:48:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:48:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:48:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:48:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:48:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:48:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:48:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:48:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:49:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:49:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:49:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:49:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:49:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:49:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:49:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:49:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:49:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:49:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:49:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:50:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:50:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:50:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:50:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:50:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:50:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:50:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:50:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:50:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:50:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:50:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:51:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:51:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:51:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:51:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:51:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:51:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:51:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:51:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:51:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:51:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:51:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:51:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:52:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:52:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:52:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:52:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:52:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:52:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:52:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:52:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:52:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:52:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:52:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:53:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:53:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:53:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:53:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:53:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:53:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:53:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:53:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:53:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:53:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:53:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:53:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:54:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:54:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:54:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:54:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:54:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:54:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:54:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:54:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:54:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:54:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:54:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 21:54:58:[cuda:0] Epoch 800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 21:55:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 21:55:04:mr: 387.71 INFO:Trainer:2024-05-23 21:55:04:mrr: 0.262 INFO:Trainer:2024-05-23 21:55:04:hits@1: 0.192 INFO:Trainer:2024-05-23 21:55:04:hits@3: 0.284 INFO:Trainer:2024-05-23 21:55:04:hits@10: 0.399 INFO:Trainer:2024-05-23 21:55:04:mr_type: 202.812 INFO:Trainer:2024-05-23 21:55:04:mrr_type: 0.286 INFO:Trainer:2024-05-23 21:55:04:hits@1_type: 0.213 INFO:Trainer:2024-05-23 21:55:04:hits@3_type: 0.308 INFO:Trainer:2024-05-23 21:55:04:hits@10_type: 0.431 INFO:EarlyStopping:2024-05-23 21:55:04:Validation score improved (0.391000 --> 0.399000). Saving model ... INFO:Trainer:2024-05-23 21:55:05:[cuda:0] Epoch [ 800/10000] | Batchsize: 1024 | loss: 0.257085 | 7.51185 seconds/epoch DEBUG:connectionpool:2024-05-23 21:55:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:55:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:55:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:55:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:55:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:55:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:55:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:55:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:55:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:55:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:55:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:56:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:56:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:56:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:56:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:56:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:56:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:56:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:56:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:56:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:56:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:56:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:57:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:57:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:57:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:57:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:57:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:57:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:57:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:57:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:57:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:57:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:57:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:57:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:58:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:58:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:58:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:58:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:58:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:58:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:58:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:58:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:58:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:58:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:58:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:59:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:59:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:59:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:59:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:59:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:59:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:59:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:59:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:59:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:59:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:59:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 21:59:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:00:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:00:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:00:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:00:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:00:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:00:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:00:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:00:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:00:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:00:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:00:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:01:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:01:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:01:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:01:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:01:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:01:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:01:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:01:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:01:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:01:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:01:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:02:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:02:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:02:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:02:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:02:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:02:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:02:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:02:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:02:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:02:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:02:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:02:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:03:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:03:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:03:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:03:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:03:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:03:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:03:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:03:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:03:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:03:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:03:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:04:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:04:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:04:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:04:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:04:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:04:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:04:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:04:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:04:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:04:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:04:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:04:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:05:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:05:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:05:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:05:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:05:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:05:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:05:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:05:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:05:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:05:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:05:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:06:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:06:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:06:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:06:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:06:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:06:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:06:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:06:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:06:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:06:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:06:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:07:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:07:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:07:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:07:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:07:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 22:07:23:[cuda:0] Epoch 900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 22:07:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 22:07:29:mr: 368.318 INFO:Trainer:2024-05-23 22:07:29:mrr: 0.266 INFO:Trainer:2024-05-23 22:07:29:hits@1: 0.194 INFO:Trainer:2024-05-23 22:07:29:hits@3: 0.288 INFO:Trainer:2024-05-23 22:07:29:hits@10: 0.405 INFO:Trainer:2024-05-23 22:07:29:mr_type: 194.003 INFO:Trainer:2024-05-23 22:07:29:mrr_type: 0.29 INFO:Trainer:2024-05-23 22:07:29:hits@1_type: 0.215 INFO:Trainer:2024-05-23 22:07:29:hits@3_type: 0.313 INFO:Trainer:2024-05-23 22:07:29:hits@10_type: 0.436 INFO:EarlyStopping:2024-05-23 22:07:29:Validation score improved (0.399000 --> 0.405000). Saving model ... INFO:Trainer:2024-05-23 22:07:29:[cuda:0] Epoch [ 900/10000] | Batchsize: 1024 | loss: 0.253934 | 7.50454 seconds/epoch DEBUG:connectionpool:2024-05-23 22:07:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:07:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:07:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:07:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:07:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:07:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:08:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:08:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:08:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:08:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:08:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:08:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:08:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:08:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:08:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:08:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:08:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:09:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:09:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:09:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:09:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:09:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:09:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:09:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:09:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:09:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:09:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:09:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:10:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:10:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:10:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:10:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:10:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:10:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:10:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:10:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:10:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:10:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:10:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:10:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:11:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:11:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:11:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:11:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:11:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:11:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:11:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:11:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:11:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:11:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:11:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:12:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:12:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:12:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:12:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:12:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:12:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:12:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:12:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:12:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:12:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:12:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:13:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:13:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:13:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:13:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:13:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:13:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:13:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:13:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:13:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:13:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:13:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:13:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:14:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:14:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:14:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:14:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:14:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:14:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:14:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:14:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:14:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:14:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:14:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:15:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:15:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:15:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:15:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:15:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:15:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:15:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:15:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:15:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:15:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:15:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:16:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:16:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:16:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:16:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:16:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:16:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:16:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:16:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:16:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:16:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:16:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:16:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:17:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:17:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:17:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:17:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:17:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:17:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:17:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:17:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:17:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:17:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:17:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:18:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:18:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:18:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:18:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:18:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:18:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:18:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:18:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:18:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:18:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:18:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:19:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:19:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:19:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:19:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:19:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:19:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:19:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:19:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:19:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 22:19:46:[cuda:0] Epoch 1000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 22:19:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 22:19:52:mr: 351.694 INFO:Trainer:2024-05-23 22:19:52:mrr: 0.27 INFO:Trainer:2024-05-23 22:19:52:hits@1: 0.198 INFO:Trainer:2024-05-23 22:19:52:hits@3: 0.293 INFO:Trainer:2024-05-23 22:19:52:hits@10: 0.411 INFO:Trainer:2024-05-23 22:19:52:mr_type: 186.272 INFO:Trainer:2024-05-23 22:19:52:mrr_type: 0.294 INFO:Trainer:2024-05-23 22:19:52:hits@1_type: 0.22 INFO:Trainer:2024-05-23 22:19:52:hits@3_type: 0.317 INFO:Trainer:2024-05-23 22:19:52:hits@10_type: 0.442 INFO:EarlyStopping:2024-05-23 22:19:52:Validation score improved (0.405000 --> 0.411000). Saving model ... INFO:Trainer:2024-05-23 22:19:52:[cuda:0] Epoch [1000/10000] | Batchsize: 1024 | loss: 0.251452 | 7.49674 seconds/epoch DEBUG:connectionpool:2024-05-23 22:19:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:19:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:20:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:20:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:20:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:20:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:20:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:20:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:20:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:20:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:20:46:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-23 22:20:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:20:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:20:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:21:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:21:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:21:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:21:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:21:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:21:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:21:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:21:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:21:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:21:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:21:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:22:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:22:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:22:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:22:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:22:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:22:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:22:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:22:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:22:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:22:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:22:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:22:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:23:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:23:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:23:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:23:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:23:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:23:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:23:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:23:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:23:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:23:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:23:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:24:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:24:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:24:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:24:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:24:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:24:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:24:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:24:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:24:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:24:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:24:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:25:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:25:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:25:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:25:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:25:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:25:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:25:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:25:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:25:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:25:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:25:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:26:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:26:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:26:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:26:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:26:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:26:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:26:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:26:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:26:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:26:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:26:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:26:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:27:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:27:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:27:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:27:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:27:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:27:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:27:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:27:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:27:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:27:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:27:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:28:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:28:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:28:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:28:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:28:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:28:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:28:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:28:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:28:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:28:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:28:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:29:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:29:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:29:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:29:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:29:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:29:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:29:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:29:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:29:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:29:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:29:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:29:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:30:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:30:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:30:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:30:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:30:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:30:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:30:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:30:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:30:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:30:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:30:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:31:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:31:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:31:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:31:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:31:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:31:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:31:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:31:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:31:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:31:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:31:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:31:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:32:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:32:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 22:32:11:[cuda:0] Epoch 1100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 22:32:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 22:32:18:mr: 340.386 INFO:Trainer:2024-05-23 22:32:18:mrr: 0.273 INFO:Trainer:2024-05-23 22:32:18:hits@1: 0.2 INFO:Trainer:2024-05-23 22:32:18:hits@3: 0.297 INFO:Trainer:2024-05-23 22:32:18:hits@10: 0.417 INFO:Trainer:2024-05-23 22:32:18:mr_type: 179.521 INFO:Trainer:2024-05-23 22:32:18:mrr_type: 0.298 INFO:Trainer:2024-05-23 22:32:18:hits@1_type: 0.222 INFO:Trainer:2024-05-23 22:32:18:hits@3_type: 0.322 INFO:Trainer:2024-05-23 22:32:18:hits@10_type: 0.447 INFO:EarlyStopping:2024-05-23 22:32:18:Validation score improved (0.411000 --> 0.417000). Saving model ... INFO:Trainer:2024-05-23 22:32:18:[cuda:0] Epoch [1100/10000] | Batchsize: 1024 | loss: 0.249159 | 7.49304 seconds/epoch DEBUG:connectionpool:2024-05-23 22:32:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:32:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:32:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:32:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:32:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:32:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:32:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:32:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:33:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:33:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:33:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:33:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:33:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:33:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:33:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:33:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:33:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:33:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:33:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:34:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:34:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:34:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:34:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:34:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:34:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:34:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:34:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:34:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:34:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:34:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:34:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:35:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:35:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:35:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:35:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:35:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:35:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:35:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:35:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:35:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:35:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:35:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:36:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:36:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:36:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:36:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:36:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:36:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:36:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:36:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:36:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:36:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:36:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:37:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:37:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:37:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:37:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:37:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:37:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:37:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:37:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:37:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:37:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:37:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:37:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:38:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:38:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:38:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:38:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:38:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:38:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:38:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:38:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:38:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:38:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:38:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:39:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:39:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:39:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:39:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:39:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:39:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:39:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:39:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:39:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:39:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:39:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:40:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:40:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:40:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:40:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:40:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:40:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:40:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:40:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:40:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:40:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:40:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:40:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:41:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:41:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:41:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:41:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:41:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:41:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:41:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:41:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:41:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:41:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:41:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:42:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:42:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:42:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:42:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:42:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:42:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:42:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:42:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:42:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:42:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:42:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:42:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:43:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:43:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:43:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:43:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:43:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:43:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:43:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:43:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:43:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:43:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:43:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:44:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:44:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:44:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:44:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:44:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:44:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:44:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:44:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 22:44:42:[cuda:0] Epoch 1200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 22:44:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 22:44:48:mr: 330.527 INFO:Trainer:2024-05-23 22:44:48:mrr: 0.275 INFO:Trainer:2024-05-23 22:44:48:hits@1: 0.201 INFO:Trainer:2024-05-23 22:44:48:hits@3: 0.3 INFO:Trainer:2024-05-23 22:44:48:hits@10: 0.422 INFO:Trainer:2024-05-23 22:44:48:mr_type: 173.372 INFO:Trainer:2024-05-23 22:44:48:mrr_type: 0.3 INFO:Trainer:2024-05-23 22:44:48:hits@1_type: 0.223 INFO:Trainer:2024-05-23 22:44:48:hits@3_type: 0.325 INFO:Trainer:2024-05-23 22:44:48:hits@10_type: 0.452 INFO:EarlyStopping:2024-05-23 22:44:48:Validation score improved (0.417000 --> 0.422000). Saving model ... INFO:Trainer:2024-05-23 22:44:48:[cuda:0] Epoch [1200/10000] | Batchsize: 1024 | loss: 0.246061 | 7.49445 seconds/epoch DEBUG:connectionpool:2024-05-23 22:44:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:44:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:45:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:45:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:45:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:45:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:45:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:45:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:45:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:45:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:45:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:45:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:45:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:45:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:46:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:46:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:46:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:46:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:46:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:46:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:46:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:46:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:46:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:46:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:46:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:47:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:47:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:47:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:47:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:47:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:47:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:47:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:47:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:47:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:47:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:47:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:47:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:48:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:48:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:48:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:48:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:48:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:48:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:48:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:48:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:48:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:48:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:48:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:49:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:49:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:49:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:49:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:49:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:49:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:49:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:49:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:49:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:49:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:49:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:50:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:50:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:50:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:50:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:50:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:50:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:50:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:50:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:50:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:50:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:50:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:50:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:51:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:51:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:51:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:51:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:51:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:51:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:51:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:51:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:51:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:51:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:51:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:52:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:52:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:52:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:52:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:52:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:52:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:52:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:52:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:52:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:52:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:52:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:53:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:53:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:53:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:53:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:53:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:53:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:53:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:53:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:53:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:53:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:53:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:53:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:54:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:54:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:54:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:54:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:54:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:54:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:54:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:54:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:54:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:54:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:54:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:55:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:55:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:55:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:55:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:55:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:55:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:55:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:55:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:55:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:55:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:55:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:55:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:56:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:56:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:56:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:56:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:56:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:56:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:56:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:56:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:56:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:56:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:56:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:57:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 22:57:06:[cuda:0] Epoch 1300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 22:57:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 22:57:12:mr: 314.842 INFO:Trainer:2024-05-23 22:57:12:mrr: 0.28 INFO:Trainer:2024-05-23 22:57:12:hits@1: 0.204 INFO:Trainer:2024-05-23 22:57:12:hits@3: 0.304 INFO:Trainer:2024-05-23 22:57:12:hits@10: 0.428 INFO:Trainer:2024-05-23 22:57:12:mr_type: 167.019 INFO:Trainer:2024-05-23 22:57:12:mrr_type: 0.304 INFO:Trainer:2024-05-23 22:57:12:hits@1_type: 0.226 INFO:Trainer:2024-05-23 22:57:12:hits@3_type: 0.329 INFO:Trainer:2024-05-23 22:57:12:hits@10_type: 0.457 INFO:EarlyStopping:2024-05-23 22:57:12:Validation score improved (0.422000 --> 0.428000). Saving model ... INFO:Trainer:2024-05-23 22:57:12:[cuda:0] Epoch [1300/10000] | Batchsize: 1024 | loss: 0.243693 | 7.49011 seconds/epoch DEBUG:connectionpool:2024-05-23 22:57:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:57:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:57:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:57:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:57:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:57:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:57:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:57:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:57:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:58:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:58:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:58:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:58:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:58:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:58:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:58:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:58:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:58:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:58:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:58:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:58:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:59:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:59:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:59:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:59:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:59:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:59:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:59:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:59:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:59:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:59:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 22:59:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:00:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:00:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:00:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:00:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:00:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:00:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:00:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:00:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:00:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:00:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:00:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:01:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:01:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:01:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:01:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:01:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:01:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:01:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:01:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:01:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:01:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:01:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:01:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:02:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:02:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:02:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:02:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:02:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:02:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:02:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:02:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:02:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:02:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:02:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:03:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:03:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:03:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:03:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:03:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:03:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:03:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:03:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:03:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:03:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:03:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:03:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:04:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:04:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:04:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:04:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:04:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:04:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:04:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:04:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:04:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:04:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:04:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:05:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:05:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:05:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:05:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:05:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:05:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:05:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:05:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:05:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:05:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:05:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:06:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:06:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:06:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:06:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:06:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:06:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:06:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:06:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:06:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:06:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:06:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:06:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:07:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:07:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:07:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:07:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:07:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:07:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:07:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:07:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:07:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:07:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:07:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:08:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:08:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:08:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:08:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:08:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:08:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:08:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:08:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:08:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:08:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:08:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:08:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:09:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:09:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:09:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:09:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:09:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:09:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 23:09:35:[cuda:0] Epoch 1400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 23:09:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:09:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 23:09:41:mr: 304.845 INFO:Trainer:2024-05-23 23:09:41:mrr: 0.283 INFO:Trainer:2024-05-23 23:09:41:hits@1: 0.207 INFO:Trainer:2024-05-23 23:09:41:hits@3: 0.307 INFO:Trainer:2024-05-23 23:09:41:hits@10: 0.432 INFO:Trainer:2024-05-23 23:09:41:mr_type: 161.439 INFO:Trainer:2024-05-23 23:09:41:mrr_type: 0.308 INFO:Trainer:2024-05-23 23:09:41:hits@1_type: 0.229 INFO:Trainer:2024-05-23 23:09:41:hits@3_type: 0.332 INFO:Trainer:2024-05-23 23:09:41:hits@10_type: 0.462 INFO:EarlyStopping:2024-05-23 23:09:41:Validation score improved (0.428000 --> 0.432000). Saving model ... INFO:Trainer:2024-05-23 23:09:41:[cuda:0] Epoch [1400/10000] | Batchsize: 1024 | loss: 0.241663 | 7.48982 seconds/epoch DEBUG:connectionpool:2024-05-23 23:09:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:09:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:09:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:10:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:10:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:10:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:10:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:10:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:10:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:10:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:10:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:10:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:10:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:10:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:10:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:11:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:11:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:11:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:11:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:11:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:11:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:11:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:11:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:11:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:11:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:11:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:12:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:12:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:12:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:12:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:12:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:12:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:12:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:12:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:12:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:12:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:12:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:13:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:13:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:13:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:13:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:13:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:13:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:13:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:13:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:13:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:13:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:13:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:13:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:14:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:14:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:14:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:14:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:14:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:14:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:14:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:14:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:14:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:14:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:14:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:15:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:15:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:15:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:15:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:15:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:15:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:15:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:15:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:15:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:15:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:15:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:16:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:16:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:16:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:16:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:16:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:16:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:16:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:16:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:16:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:16:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:16:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:16:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:17:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:17:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:17:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:17:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:17:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:17:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:17:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:17:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:17:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:17:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:17:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:18:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:18:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:18:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:18:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:18:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:18:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:18:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:18:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:18:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:18:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:18:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:18:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:19:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:19:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:19:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:19:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:19:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:19:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:19:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:19:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:19:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:19:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:19:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:20:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:20:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:20:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:20:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:20:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:20:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:20:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:20:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:20:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:20:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:20:54:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-23 23:20:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:21:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:21:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:21:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:21:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:21:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:21:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:21:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:21:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:21:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:21:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:21:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:21:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:22:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 23:22:04:[cuda:0] Epoch 1500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 23:22:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 23:22:10:mr: 298.541 INFO:Trainer:2024-05-23 23:22:10:mrr: 0.285 INFO:Trainer:2024-05-23 23:22:10:hits@1: 0.208 INFO:Trainer:2024-05-23 23:22:10:hits@3: 0.311 INFO:Trainer:2024-05-23 23:22:10:hits@10: 0.436 INFO:Trainer:2024-05-23 23:22:10:mr_type: 157.046 INFO:Trainer:2024-05-23 23:22:10:mrr_type: 0.309 INFO:Trainer:2024-05-23 23:22:10:hits@1_type: 0.23 INFO:Trainer:2024-05-23 23:22:10:hits@3_type: 0.335 INFO:Trainer:2024-05-23 23:22:10:hits@10_type: 0.466 INFO:EarlyStopping:2024-05-23 23:22:10:Validation score improved (0.432000 --> 0.436000). Saving model ... INFO:Trainer:2024-05-23 23:22:10:[cuda:0] Epoch [1500/10000] | Batchsize: 1024 | loss: 0.239841 | 7.49003 seconds/epoch DEBUG:connectionpool:2024-05-23 23:22:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:22:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:22:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:22:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:22:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:22:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:22:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:22:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:22:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:23:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:23:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:23:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:23:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:23:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:23:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:23:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:23:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:23:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:23:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:23:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:23:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:24:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:24:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:24:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:24:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:24:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:24:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:24:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:24:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:24:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:24:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:24:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:25:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:25:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:25:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:25:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:25:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:25:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:25:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:25:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:25:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:25:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:25:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:25:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:26:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:26:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:26:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:26:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:26:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:26:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:26:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:26:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:26:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:26:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:26:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:27:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:27:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:27:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:27:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:27:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:27:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:27:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:27:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:27:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:27:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:27:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:27:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:28:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:28:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:28:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:28:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:28:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:28:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:28:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:28:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:28:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:28:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:28:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:29:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:29:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:29:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:29:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:29:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:29:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:29:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:29:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:29:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:29:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:29:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:30:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:30:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:30:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:30:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:30:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:30:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:30:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:30:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:30:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:30:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:30:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:31:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:31:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:31:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:31:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:31:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:31:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:31:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:31:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:31:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:31:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:31:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:32:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:32:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:32:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:32:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:32:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:32:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:32:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:32:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:32:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:32:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:32:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:32:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:33:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:33:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:33:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:33:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:33:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:33:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:33:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:33:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:33:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:33:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:33:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:34:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:34:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:34:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:34:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:34:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:34:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 23:34:34:[cuda:0] Epoch 1600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 23:34:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 23:34:39:mr: 288.357 INFO:Trainer:2024-05-23 23:34:39:mrr: 0.288 INFO:Trainer:2024-05-23 23:34:39:hits@1: 0.211 INFO:Trainer:2024-05-23 23:34:39:hits@3: 0.313 INFO:Trainer:2024-05-23 23:34:39:hits@10: 0.439 INFO:Trainer:2024-05-23 23:34:39:mr_type: 153.632 INFO:Trainer:2024-05-23 23:34:39:mrr_type: 0.312 INFO:Trainer:2024-05-23 23:34:39:hits@1_type: 0.233 INFO:Trainer:2024-05-23 23:34:39:hits@3_type: 0.336 INFO:Trainer:2024-05-23 23:34:39:hits@10_type: 0.47 INFO:EarlyStopping:2024-05-23 23:34:39:Validation score improved (0.436000 --> 0.439000). Saving model ... INFO:Trainer:2024-05-23 23:34:40:[cuda:0] Epoch [1600/10000] | Batchsize: 1024 | loss: 0.238235 | 7.49047 seconds/epoch DEBUG:connectionpool:2024-05-23 23:34:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:34:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:34:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:34:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:35:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:35:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:35:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:35:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:35:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:35:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:35:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:35:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:35:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:35:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:35:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:36:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:36:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:36:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:36:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:36:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:36:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:36:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:36:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:36:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:36:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:36:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:37:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:37:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:37:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:37:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:37:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:37:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:37:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:37:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:37:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:37:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:37:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:38:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:38:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:38:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:38:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:38:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:38:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:38:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:38:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:38:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:38:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:38:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:38:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:39:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:39:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:39:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:39:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:39:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:39:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:39:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:39:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:39:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:39:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:39:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:40:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:40:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:40:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:40:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:40:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:40:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:40:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:40:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:40:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:40:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:40:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:41:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:41:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:41:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:41:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:41:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:41:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:41:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:41:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:41:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:41:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:41:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:42:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:42:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:42:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:42:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:42:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:42:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:42:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:42:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:42:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:42:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:42:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:42:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:43:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:43:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:43:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:43:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:43:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:43:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:43:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:43:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:43:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:43:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:43:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:44:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:44:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:44:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:44:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:44:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:44:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:44:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:44:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:44:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:44:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:44:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:45:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:45:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:45:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:45:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:45:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:45:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:45:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:45:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:45:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:45:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:45:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:46:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:46:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:46:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:46:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:46:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:46:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:46:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:46:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:46:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:46:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 23:46:56:[cuda:0] Epoch 1700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 23:46:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:47:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 23:47:02:mr: 284.63 INFO:Trainer:2024-05-23 23:47:02:mrr: 0.291 INFO:Trainer:2024-05-23 23:47:02:hits@1: 0.213 INFO:Trainer:2024-05-23 23:47:02:hits@3: 0.316 INFO:Trainer:2024-05-23 23:47:02:hits@10: 0.443 INFO:Trainer:2024-05-23 23:47:02:mr_type: 150.359 INFO:Trainer:2024-05-23 23:47:02:mrr_type: 0.314 INFO:Trainer:2024-05-23 23:47:02:hits@1_type: 0.235 INFO:Trainer:2024-05-23 23:47:02:hits@3_type: 0.34 INFO:Trainer:2024-05-23 23:47:02:hits@10_type: 0.473 INFO:EarlyStopping:2024-05-23 23:47:02:Validation score improved (0.439000 --> 0.443000). Saving model ... INFO:Trainer:2024-05-23 23:47:02:[cuda:0] Epoch [1700/10000] | Batchsize: 1024 | loss: 0.236962 | 7.48658 seconds/epoch DEBUG:connectionpool:2024-05-23 23:47:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:47:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:47:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:47:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:47:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:47:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:47:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:47:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:47:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:47:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:48:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:48:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:48:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:48:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:48:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:48:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:48:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:48:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:48:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:48:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:48:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:48:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:49:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:49:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:49:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:49:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:49:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:49:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:49:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:49:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:49:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:49:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:49:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:50:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:50:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:50:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:50:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:50:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:50:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:50:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:50:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:50:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:50:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:50:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:51:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:51:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:51:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:51:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:51:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:51:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:51:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:51:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:51:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:51:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:51:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:51:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:52:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:52:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:52:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:52:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:52:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:52:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:52:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:52:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:52:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:52:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:52:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:53:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:53:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:53:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:53:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:53:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:53:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:53:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:53:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:53:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:53:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:54:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:54:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:54:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:54:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:54:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:54:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:54:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:54:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:54:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:54:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:54:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:55:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:55:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:55:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:55:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:55:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:55:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:55:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:55:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:55:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:55:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:55:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:56:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:56:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:56:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:56:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:56:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:56:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:56:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:56:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:56:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:56:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:56:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:57:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:57:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:57:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:57:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:57:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:57:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:57:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:57:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:57:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:57:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:57:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:58:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:58:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:58:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:58:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:58:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:58:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:58:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:58:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:58:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:58:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:58:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:59:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:59:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:59:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:59:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 23:59:20:[cuda:0] Epoch 1800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-23 23:59:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-23 23:59:26:mr: 277.008 INFO:Trainer:2024-05-23 23:59:26:mrr: 0.293 INFO:Trainer:2024-05-23 23:59:26:hits@1: 0.215 INFO:Trainer:2024-05-23 23:59:26:hits@3: 0.318 INFO:Trainer:2024-05-23 23:59:26:hits@10: 0.447 INFO:Trainer:2024-05-23 23:59:26:mr_type: 147.28 INFO:Trainer:2024-05-23 23:59:26:mrr_type: 0.317 INFO:Trainer:2024-05-23 23:59:26:hits@1_type: 0.237 INFO:Trainer:2024-05-23 23:59:26:hits@3_type: 0.342 INFO:Trainer:2024-05-23 23:59:26:hits@10_type: 0.477 INFO:EarlyStopping:2024-05-23 23:59:26:Validation score improved (0.443000 --> 0.447000). Saving model ... INFO:Trainer:2024-05-23 23:59:26:[cuda:0] Epoch [1800/10000] | Batchsize: 1024 | loss: 0.235682 | 7.48388 seconds/epoch DEBUG:connectionpool:2024-05-23 23:59:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:59:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:59:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:59:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:59:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:59:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-23 23:59:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:00:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:00:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:00:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:00:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:00:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:00:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:00:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:00:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:00:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:00:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:00:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:01:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:01:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:01:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:01:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:01:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:01:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:01:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:01:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:01:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:01:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:01:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:01:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:02:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:02:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:02:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:02:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:02:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:02:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:02:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:02:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:02:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:02:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:02:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:03:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:03:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:03:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:03:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:03:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:03:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:03:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:03:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:03:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:03:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:03:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:04:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:04:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:04:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:04:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:04:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:04:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:04:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:04:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:04:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:04:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:04:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:04:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:05:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:05:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:05:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:05:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:05:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:05:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:05:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:05:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:05:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:05:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:05:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:06:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:06:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:06:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:06:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:06:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:06:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:06:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:06:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:06:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:06:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:06:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:06:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:07:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:07:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:07:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:07:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:07:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:07:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:07:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:07:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:07:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:07:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:07:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:08:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:08:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:08:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:08:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:08:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:08:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:08:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:08:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:08:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:08:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:08:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:09:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:09:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:09:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:09:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:09:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:09:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:09:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:09:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:09:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:09:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:09:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:09:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:10:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:10:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:10:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:10:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:10:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:10:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:10:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:10:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:10:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:10:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:10:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:11:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:11:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:11:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:11:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:11:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:11:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:11:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:11:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:11:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:11:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 00:11:48:[cuda:0] Epoch 1900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 00:11:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 00:11:54:mr: 272.083 INFO:Trainer:2024-05-24 00:11:54:mrr: 0.295 INFO:Trainer:2024-05-24 00:11:54:hits@1: 0.217 INFO:Trainer:2024-05-24 00:11:54:hits@3: 0.322 INFO:Trainer:2024-05-24 00:11:54:hits@10: 0.45 INFO:Trainer:2024-05-24 00:11:54:mr_type: 144.234 INFO:Trainer:2024-05-24 00:11:54:mrr_type: 0.319 INFO:Trainer:2024-05-24 00:11:54:hits@1_type: 0.238 INFO:Trainer:2024-05-24 00:11:54:hits@3_type: 0.346 INFO:Trainer:2024-05-24 00:11:54:hits@10_type: 0.48 INFO:EarlyStopping:2024-05-24 00:11:54:Validation score improved (0.447000 --> 0.450000). Saving model ... INFO:Trainer:2024-05-24 00:11:54:[cuda:0] Epoch [1900/10000] | Batchsize: 1024 | loss: 0.234644 | 7.48369 seconds/epoch DEBUG:connectionpool:2024-05-24 00:11:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:12:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:12:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:12:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:12:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:12:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:12:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:12:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:12:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:12:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:12:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:12:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:13:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:13:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:13:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:13:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:13:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:13:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:13:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:13:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:13:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:13:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:13:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:13:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:14:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:14:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:14:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:14:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:14:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:14:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:14:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:14:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:14:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:14:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:14:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:15:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:15:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:15:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:15:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:15:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:15:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:15:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:15:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:15:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:15:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:15:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:15:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:16:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:16:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:16:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:16:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:16:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:16:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:16:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:16:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:16:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:16:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:16:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:17:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:17:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:17:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:17:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:17:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:17:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:17:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:17:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:17:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:17:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:17:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:17:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:18:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:18:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:18:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:18:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:18:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:18:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:18:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:18:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:18:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:18:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:18:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:19:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:19:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:19:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:19:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:19:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:19:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:19:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:19:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:19:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:19:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:19:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:20:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:20:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:20:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:20:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:20:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:20:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:20:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:20:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:20:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:20:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:20:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:20:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:21:04:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-24 00:21:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:21:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:21:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:21:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:21:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:21:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:21:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:21:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:21:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:21:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:21:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:22:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:22:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:22:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:22:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:22:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:22:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:22:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:22:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:22:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:22:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:22:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:23:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:23:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:23:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:23:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:23:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:23:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:23:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:23:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:23:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:23:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:23:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:23:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:24:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:24:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:24:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 00:24:17:[cuda:0] Epoch 2000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 00:24:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 00:24:23:mr: 267.346 INFO:Trainer:2024-05-24 00:24:23:mrr: 0.297 INFO:Trainer:2024-05-24 00:24:23:hits@1: 0.219 INFO:Trainer:2024-05-24 00:24:23:hits@3: 0.323 INFO:Trainer:2024-05-24 00:24:23:hits@10: 0.453 INFO:Trainer:2024-05-24 00:24:23:mr_type: 141.875 INFO:Trainer:2024-05-24 00:24:23:mrr_type: 0.321 INFO:Trainer:2024-05-24 00:24:23:hits@1_type: 0.24 INFO:Trainer:2024-05-24 00:24:23:hits@3_type: 0.348 INFO:Trainer:2024-05-24 00:24:23:hits@10_type: 0.483 INFO:EarlyStopping:2024-05-24 00:24:23:Validation score improved (0.450000 --> 0.453000). Saving model ... INFO:Trainer:2024-05-24 00:24:23:[cuda:0] Epoch [2000/10000] | Batchsize: 1024 | loss: 0.233637 | 7.48403 seconds/epoch DEBUG:connectionpool:2024-05-24 00:24:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:24:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:24:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:24:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:24:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:24:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:24:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:25:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:25:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:25:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:25:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:25:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:25:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:25:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:25:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:25:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:25:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:25:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:26:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:26:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:26:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:26:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:26:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:26:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:26:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:26:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:26:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:26:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:26:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:26:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:27:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:27:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:27:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:27:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:27:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:27:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:27:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:27:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:27:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:27:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:27:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:28:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:28:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:28:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:28:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:28:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:28:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:28:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:28:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:28:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:28:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:28:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:28:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:29:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:29:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:29:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:29:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:29:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:29:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:29:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:29:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:29:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:29:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:29:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:30:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:30:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:30:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:30:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:30:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:30:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:30:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:30:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:30:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:30:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:30:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:30:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:31:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:31:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:31:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:31:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:31:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:31:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:31:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:31:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:31:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:31:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:31:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:32:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:32:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:32:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:32:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:32:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:32:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:32:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:32:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:32:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:32:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:32:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:32:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:33:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:33:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:33:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:33:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:33:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:33:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:33:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:33:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:33:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:33:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:33:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:34:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:34:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:34:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:34:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:34:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:34:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:34:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:34:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:34:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:34:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:34:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:35:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:35:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:35:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:35:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:35:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:35:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:35:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:35:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:35:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:35:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:35:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:35:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:36:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:36:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:36:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:36:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:36:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:36:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:36:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:36:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 00:36:41:[cuda:0] Epoch 2100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 00:36:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 00:36:47:mr: 264.66 INFO:Trainer:2024-05-24 00:36:47:mrr: 0.3 INFO:Trainer:2024-05-24 00:36:47:hits@1: 0.22 INFO:Trainer:2024-05-24 00:36:47:hits@3: 0.326 INFO:Trainer:2024-05-24 00:36:47:hits@10: 0.455 INFO:Trainer:2024-05-24 00:36:47:mr_type: 139.734 INFO:Trainer:2024-05-24 00:36:47:mrr_type: 0.324 INFO:Trainer:2024-05-24 00:36:47:hits@1_type: 0.242 INFO:Trainer:2024-05-24 00:36:47:hits@3_type: 0.351 INFO:Trainer:2024-05-24 00:36:47:hits@10_type: 0.486 INFO:EarlyStopping:2024-05-24 00:36:47:Validation score improved (0.453000 --> 0.455000). Saving model ... INFO:Trainer:2024-05-24 00:36:47:[cuda:0] Epoch [2100/10000] | Batchsize: 1024 | loss: 0.232985 | 7.48215 seconds/epoch DEBUG:connectionpool:2024-05-24 00:36:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:36:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:37:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:37:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:37:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:37:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:37:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:37:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:37:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:37:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:37:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:37:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:37:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:37:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:38:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:38:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:38:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:38:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:38:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:38:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:38:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:38:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:38:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:38:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:38:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:39:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:39:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:39:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:39:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:39:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:39:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:39:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:39:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:39:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:39:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:39:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:40:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:40:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:40:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:40:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:40:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:40:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:40:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:40:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:40:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:40:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:40:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:40:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:41:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:41:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:41:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:41:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:41:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:41:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:41:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:41:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:41:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:41:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:41:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:42:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:42:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:42:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:42:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:42:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:42:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:42:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:42:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:42:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:42:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:42:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:42:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:43:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:43:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:43:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:43:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:43:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:43:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:43:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:43:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:43:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:43:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:43:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:44:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:44:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:44:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:44:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:44:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:44:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:44:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:44:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:44:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:44:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:44:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:44:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:45:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:45:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:45:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:45:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:45:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:45:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:45:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:45:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:45:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:45:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:45:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:46:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:46:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:46:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:46:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:46:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:46:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:46:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:46:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:46:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:46:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:46:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:47:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:47:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:47:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:47:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:47:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:47:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:47:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:47:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:47:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:47:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:47:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:47:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:48:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:48:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:48:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:48:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:48:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:48:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:48:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:48:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:48:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:48:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:48:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:49:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 00:49:02:[cuda:0] Epoch 2200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 00:49:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 00:49:09:mr: 263.676 INFO:Trainer:2024-05-24 00:49:09:mrr: 0.301 INFO:Trainer:2024-05-24 00:49:09:hits@1: 0.221 INFO:Trainer:2024-05-24 00:49:09:hits@3: 0.328 INFO:Trainer:2024-05-24 00:49:09:hits@10: 0.458 INFO:Trainer:2024-05-24 00:49:09:mr_type: 137.989 INFO:Trainer:2024-05-24 00:49:09:mrr_type: 0.325 INFO:Trainer:2024-05-24 00:49:09:hits@1_type: 0.242 INFO:Trainer:2024-05-24 00:49:09:hits@3_type: 0.352 INFO:Trainer:2024-05-24 00:49:09:hits@10_type: 0.489 INFO:EarlyStopping:2024-05-24 00:49:09:Validation score improved (0.455000 --> 0.458000). Saving model ... INFO:Trainer:2024-05-24 00:49:09:[cuda:0] Epoch [2200/10000] | Batchsize: 1024 | loss: 0.231990 | 7.47891 seconds/epoch DEBUG:connectionpool:2024-05-24 00:49:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:49:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:49:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:49:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:49:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:49:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:49:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:49:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:49:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:49:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:50:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:50:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:50:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:50:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:50:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:50:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:50:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:50:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:50:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:50:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:50:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:51:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:51:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:51:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:51:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:51:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:51:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:51:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:51:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:51:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:51:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:51:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:51:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:52:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:52:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:52:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:52:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:52:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:52:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:52:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:52:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:52:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:52:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:52:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:53:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:53:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:53:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:53:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:53:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:53:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:53:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:53:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:53:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:53:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:53:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:53:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:54:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:54:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:54:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:54:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:54:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:54:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:54:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:54:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:54:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:54:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:54:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:55:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:55:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:55:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:55:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:55:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:55:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:55:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:55:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:55:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:55:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:55:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:55:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:56:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:56:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:56:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:56:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:56:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:56:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:56:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:56:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:56:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:56:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:56:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:57:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:57:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:57:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:57:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:57:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:57:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:57:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:57:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:57:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:57:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:57:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:58:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:58:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:58:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:58:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:58:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:58:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:58:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:58:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:58:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:58:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:58:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:58:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:59:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:59:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:59:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:59:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:59:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:59:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:59:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:59:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:59:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:59:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 00:59:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:00:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:00:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:00:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:00:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:00:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:00:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:00:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:00:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:00:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:00:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:00:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:00:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:01:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:01:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:01:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:01:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:01:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 01:01:26:[cuda:0] Epoch 2300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 01:01:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 01:01:32:mr: 257.522 INFO:Trainer:2024-05-24 01:01:32:mrr: 0.302 INFO:Trainer:2024-05-24 01:01:32:hits@1: 0.222 INFO:Trainer:2024-05-24 01:01:32:hits@3: 0.33 INFO:Trainer:2024-05-24 01:01:32:hits@10: 0.46 INFO:Trainer:2024-05-24 01:01:32:mr_type: 135.498 INFO:Trainer:2024-05-24 01:01:32:mrr_type: 0.327 INFO:Trainer:2024-05-24 01:01:32:hits@1_type: 0.243 INFO:Trainer:2024-05-24 01:01:32:hits@3_type: 0.354 INFO:Trainer:2024-05-24 01:01:32:hits@10_type: 0.491 INFO:EarlyStopping:2024-05-24 01:01:32:Validation score improved (0.458000 --> 0.460000). Saving model ... INFO:Trainer:2024-05-24 01:01:32:[cuda:0] Epoch [2300/10000] | Batchsize: 1024 | loss: 0.230961 | 7.47701 seconds/epoch DEBUG:connectionpool:2024-05-24 01:01:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:01:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:01:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:01:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:01:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:02:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:02:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:02:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:02:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:02:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:02:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:02:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:02:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:02:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:02:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:02:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:03:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:03:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:03:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:03:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:03:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:03:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:03:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:03:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:03:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:03:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:03:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:03:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:04:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:04:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:04:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:04:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:04:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:04:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:04:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:04:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:04:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:04:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:04:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:05:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:05:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:05:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:05:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:05:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:05:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:05:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:05:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:05:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:05:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:05:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:05:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:06:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:06:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:06:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:06:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:06:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:06:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:06:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:06:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:06:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:06:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:06:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:07:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:07:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:07:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:07:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:07:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:07:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:07:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:07:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:07:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:07:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:07:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:08:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:08:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:08:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:08:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:08:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:08:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:08:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:08:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:08:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:08:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:08:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:08:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:09:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:09:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:09:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:09:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:09:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:09:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:09:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:09:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:09:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:09:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:09:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:10:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:10:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:10:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:10:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:10:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:10:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:10:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:10:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:10:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:10:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:10:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:10:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:11:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:11:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:11:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:11:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:11:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:11:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:11:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:11:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:11:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:11:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:11:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:12:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:12:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:12:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:12:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:12:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:12:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:12:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:12:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:12:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:12:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:12:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:13:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:13:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:13:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:13:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:13:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:13:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:13:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:13:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:13:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:13:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:13:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 01:13:54:[cuda:0] Epoch 2400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 01:13:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 01:14:00:mr: 255.024 INFO:Trainer:2024-05-24 01:14:00:mrr: 0.304 INFO:Trainer:2024-05-24 01:14:00:hits@1: 0.223 INFO:Trainer:2024-05-24 01:14:00:hits@3: 0.332 INFO:Trainer:2024-05-24 01:14:00:hits@10: 0.462 INFO:Trainer:2024-05-24 01:14:00:mr_type: 133.63 INFO:Trainer:2024-05-24 01:14:00:mrr_type: 0.328 INFO:Trainer:2024-05-24 01:14:00:hits@1_type: 0.245 INFO:Trainer:2024-05-24 01:14:00:hits@3_type: 0.356 INFO:Trainer:2024-05-24 01:14:00:hits@10_type: 0.493 INFO:EarlyStopping:2024-05-24 01:14:00:Validation score improved (0.460000 --> 0.462000). Saving model ... INFO:Trainer:2024-05-24 01:14:00:[cuda:0] Epoch [2400/10000] | Batchsize: 1024 | loss: 0.230485 | 7.47705 seconds/epoch DEBUG:connectionpool:2024-05-24 01:14:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:14:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:14:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:14:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:14:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:14:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:14:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:14:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:14:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:14:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:14:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:15:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:15:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:15:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:15:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:15:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:15:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:15:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:15:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:15:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:15:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:15:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:15:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:16:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:16:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:16:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:16:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:16:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:16:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:16:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:16:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:16:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:16:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:16:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:17:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:17:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:17:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:17:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:17:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:17:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:17:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:17:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:17:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:17:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:17:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:18:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:18:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:18:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:18:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:18:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:18:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:18:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:18:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:18:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:18:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:18:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:18:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:19:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:19:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:19:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:19:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:19:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:19:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:19:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:19:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:19:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:19:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:19:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:20:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:20:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:20:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:20:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:20:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:20:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:20:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:20:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:20:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:20:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:20:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:20:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:21:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:21:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:21:14:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-24 01:21:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:21:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:21:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:21:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:21:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:21:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:21:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:21:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:21:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:22:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:22:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:22:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:22:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:22:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:22:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:22:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:22:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:22:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:22:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:22:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:22:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:23:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:23:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:23:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:23:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:23:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:23:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:23:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:23:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:23:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:23:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:23:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:24:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:24:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:24:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:24:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:24:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:24:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:24:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:24:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:24:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:24:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:24:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:25:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:25:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:25:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:25:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:25:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:25:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:25:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:25:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:25:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:25:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:25:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:25:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:26:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:26:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:26:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:26:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 01:26:20:[cuda:0] Epoch 2500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 01:26:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 01:26:26:mr: 252.544 INFO:Trainer:2024-05-24 01:26:26:mrr: 0.306 INFO:Trainer:2024-05-24 01:26:26:hits@1: 0.225 INFO:Trainer:2024-05-24 01:26:26:hits@3: 0.333 INFO:Trainer:2024-05-24 01:26:26:hits@10: 0.464 INFO:Trainer:2024-05-24 01:26:26:mr_type: 131.717 INFO:Trainer:2024-05-24 01:26:26:mrr_type: 0.331 INFO:Trainer:2024-05-24 01:26:26:hits@1_type: 0.247 INFO:Trainer:2024-05-24 01:26:26:hits@3_type: 0.357 INFO:Trainer:2024-05-24 01:26:26:hits@10_type: 0.496 INFO:EarlyStopping:2024-05-24 01:26:26:Validation score improved (0.462000 --> 0.464000). Saving model ... INFO:Trainer:2024-05-24 01:26:26:[cuda:0] Epoch [2500/10000] | Batchsize: 1024 | loss: 0.229591 | 7.47662 seconds/epoch DEBUG:connectionpool:2024-05-24 01:26:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:26:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:26:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:26:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:26:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:26:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:27:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:27:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:27:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:27:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:27:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:27:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:27:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:27:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:27:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:27:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:27:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:27:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:28:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:28:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:28:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:28:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:28:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:28:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:28:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:28:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:28:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:28:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:28:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:29:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:29:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:29:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:29:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:29:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:29:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:29:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:29:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:29:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:29:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:29:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:30:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:30:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:30:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:30:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:30:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:30:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:30:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:30:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:30:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:30:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:30:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:30:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:31:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:31:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:31:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:31:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:31:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:31:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:31:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:31:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:31:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:31:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:31:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:32:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:32:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:32:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:32:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:32:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:32:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:32:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:32:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:32:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:32:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:32:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:32:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:33:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:33:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:33:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:33:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:33:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:33:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:33:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:33:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:33:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:33:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:33:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:34:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:34:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:34:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:34:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:34:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:34:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:34:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:34:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:34:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:34:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:34:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:34:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:35:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:35:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:35:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:35:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:35:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:35:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:35:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:35:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:35:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:35:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:35:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:36:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:36:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:36:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:36:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:36:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:36:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:36:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:36:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:36:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:36:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:36:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:36:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:37:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:37:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:37:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:37:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:37:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:37:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:37:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:37:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:37:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:37:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:37:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:38:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:38:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:38:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:38:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:38:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:38:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:38:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:38:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:38:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 01:38:48:[cuda:0] Epoch 2600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 01:38:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 01:38:54:mr: 249.121 INFO:Trainer:2024-05-24 01:38:54:mrr: 0.308 INFO:Trainer:2024-05-24 01:38:54:hits@1: 0.226 INFO:Trainer:2024-05-24 01:38:54:hits@3: 0.335 INFO:Trainer:2024-05-24 01:38:54:hits@10: 0.467 INFO:Trainer:2024-05-24 01:38:54:mr_type: 129.663 INFO:Trainer:2024-05-24 01:38:54:mrr_type: 0.332 INFO:Trainer:2024-05-24 01:38:54:hits@1_type: 0.249 INFO:Trainer:2024-05-24 01:38:54:hits@3_type: 0.359 INFO:Trainer:2024-05-24 01:38:54:hits@10_type: 0.499 INFO:EarlyStopping:2024-05-24 01:38:54:Validation score improved (0.464000 --> 0.467000). Saving model ... INFO:Trainer:2024-05-24 01:38:54:[cuda:0] Epoch [2600/10000] | Batchsize: 1024 | loss: 0.228831 | 7.47663 seconds/epoch DEBUG:connectionpool:2024-05-24 01:38:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:39:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:39:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:39:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:39:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:39:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:39:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:39:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:39:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:39:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:39:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:39:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:39:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:40:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:40:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:40:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:40:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:40:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:40:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:40:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:40:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:40:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:40:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:40:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:41:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:41:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:41:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:41:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:41:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:41:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:41:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:41:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:41:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:41:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:41:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:41:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:42:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:42:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:42:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:42:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:42:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:42:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:42:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:42:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:42:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:42:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:42:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:43:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:43:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:43:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:43:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:43:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:43:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:43:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:43:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:43:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:43:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:43:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:44:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:44:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:44:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:44:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:44:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:44:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:44:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:44:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:44:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:44:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:44:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:44:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:45:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:45:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:45:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:45:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:45:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:45:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:45:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:45:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:45:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:45:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:45:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:46:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:46:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:46:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:46:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:46:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:46:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:46:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:46:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:46:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:46:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:46:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:46:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:47:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:47:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:47:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:47:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:47:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:47:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:47:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:47:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:47:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:47:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:47:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:48:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:48:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:48:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:48:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:48:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:48:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:48:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:48:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:48:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:48:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:48:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:48:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:49:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:49:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:49:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:49:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:49:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:49:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:49:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:49:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:49:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:49:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:49:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:50:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:50:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:50:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:50:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:50:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:50:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:50:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:50:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:50:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:50:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:50:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:51:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:51:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:51:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 01:51:11:[cuda:0] Epoch 2700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 01:51:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 01:51:18:mr: 247.806 INFO:Trainer:2024-05-24 01:51:18:mrr: 0.308 INFO:Trainer:2024-05-24 01:51:18:hits@1: 0.227 INFO:Trainer:2024-05-24 01:51:18:hits@3: 0.336 INFO:Trainer:2024-05-24 01:51:18:hits@10: 0.468 INFO:Trainer:2024-05-24 01:51:18:mr_type: 128.645 INFO:Trainer:2024-05-24 01:51:18:mrr_type: 0.333 INFO:Trainer:2024-05-24 01:51:18:hits@1_type: 0.249 INFO:Trainer:2024-05-24 01:51:18:hits@3_type: 0.361 INFO:Trainer:2024-05-24 01:51:18:hits@10_type: 0.5 INFO:EarlyStopping:2024-05-24 01:51:18:Validation score improved (0.467000 --> 0.468000). Saving model ... INFO:Trainer:2024-05-24 01:51:18:[cuda:0] Epoch [2700/10000] | Batchsize: 1024 | loss: 0.228172 | 7.47506 seconds/epoch DEBUG:connectionpool:2024-05-24 01:51:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:51:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:51:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:51:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:51:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:51:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:51:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:51:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:52:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:52:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:52:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:52:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:52:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:52:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:52:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:52:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:52:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:52:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:52:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:53:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:53:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:53:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:53:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:53:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:53:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:53:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:53:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:53:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:53:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:53:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:53:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:54:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:54:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:54:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:54:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:54:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:54:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:54:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:54:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:54:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:54:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:54:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:55:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:55:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:55:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:55:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:55:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:55:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:55:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:55:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:55:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:55:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:55:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:56:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:56:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:56:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:56:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:56:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:56:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:56:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:56:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:56:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:56:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:56:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:56:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:57:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:57:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:57:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:57:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:57:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:57:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:57:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:57:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:57:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:57:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:57:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:58:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:58:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:58:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:58:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:58:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:58:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:58:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:58:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:58:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:58:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:58:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:58:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:59:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:59:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:59:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:59:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:59:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:59:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:59:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:59:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:59:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:59:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 01:59:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:00:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:00:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:00:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:00:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:00:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:00:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:00:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:00:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:00:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:00:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:00:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:01:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:01:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:01:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:01:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:01:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:01:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:01:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:01:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:01:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:01:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:01:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:01:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:02:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:02:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:02:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:02:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:02:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:02:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:02:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:02:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:02:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:02:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:02:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:03:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:03:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:03:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:03:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:03:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:03:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:03:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 02:03:35:[cuda:0] Epoch 2800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 02:03:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 02:03:41:mr: 244.719 INFO:Trainer:2024-05-24 02:03:41:mrr: 0.31 INFO:Trainer:2024-05-24 02:03:41:hits@1: 0.228 INFO:Trainer:2024-05-24 02:03:41:hits@3: 0.338 INFO:Trainer:2024-05-24 02:03:41:hits@10: 0.471 INFO:Trainer:2024-05-24 02:03:41:mr_type: 126.862 INFO:Trainer:2024-05-24 02:03:41:mrr_type: 0.334 INFO:Trainer:2024-05-24 02:03:41:hits@1_type: 0.25 INFO:Trainer:2024-05-24 02:03:41:hits@3_type: 0.363 INFO:Trainer:2024-05-24 02:03:41:hits@10_type: 0.501 INFO:EarlyStopping:2024-05-24 02:03:41:Validation score improved (0.468000 --> 0.471000). Saving model ... INFO:Trainer:2024-05-24 02:03:41:[cuda:0] Epoch [2800/10000] | Batchsize: 1024 | loss: 0.227480 | 7.47364 seconds/epoch DEBUG:connectionpool:2024-05-24 02:03:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:03:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:03:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:03:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:04:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:04:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:04:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:04:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:04:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:04:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:04:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:04:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:04:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:04:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:04:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:05:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:05:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:05:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:05:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:05:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:05:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:05:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:05:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:05:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:05:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:05:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:06:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:06:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:06:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:06:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:06:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:06:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:06:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:06:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:06:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:06:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:06:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:06:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:07:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:07:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:07:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:07:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:07:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:07:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:07:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:07:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:07:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:07:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:07:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:08:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:08:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:08:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:08:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:08:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:08:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:08:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:08:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:08:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:08:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:08:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:08:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:09:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:09:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:09:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:09:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:09:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:09:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:09:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:09:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:09:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:09:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:09:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:10:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:10:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:10:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:10:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:10:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:10:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:10:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:10:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:10:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:10:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:10:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:10:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:11:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:11:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:11:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:11:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:11:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:11:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:11:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:11:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:11:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:11:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:11:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:12:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:12:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:12:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:12:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:12:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:12:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:12:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:12:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:12:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:12:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:12:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:12:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:13:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:13:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:13:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:13:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:13:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:13:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:13:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:13:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:13:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:13:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:13:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:14:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:14:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:14:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:14:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:14:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:14:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:14:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:14:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:14:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:14:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:14:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:14:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:15:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:15:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:15:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:15:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:15:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:15:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:15:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:15:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:15:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:15:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:15:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 02:15:59:[cuda:0] Epoch 2900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 02:16:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 02:16:06:mr: 244.184 INFO:Trainer:2024-05-24 02:16:06:mrr: 0.311 INFO:Trainer:2024-05-24 02:16:06:hits@1: 0.229 INFO:Trainer:2024-05-24 02:16:06:hits@3: 0.339 INFO:Trainer:2024-05-24 02:16:06:hits@10: 0.472 INFO:Trainer:2024-05-24 02:16:06:mr_type: 126.031 INFO:Trainer:2024-05-24 02:16:06:mrr_type: 0.336 INFO:Trainer:2024-05-24 02:16:06:hits@1_type: 0.251 INFO:Trainer:2024-05-24 02:16:06:hits@3_type: 0.364 INFO:Trainer:2024-05-24 02:16:06:hits@10_type: 0.503 INFO:EarlyStopping:2024-05-24 02:16:06:Validation score improved (0.471000 --> 0.472000). Saving model ... INFO:Trainer:2024-05-24 02:16:06:[cuda:0] Epoch [2900/10000] | Batchsize: 1024 | loss: 0.226793 | 7.47257 seconds/epoch DEBUG:connectionpool:2024-05-24 02:16:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:16:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:16:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:16:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:16:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:16:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:16:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:16:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:16:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:16:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:16:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:17:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:17:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:17:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:17:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:17:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:17:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:17:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:17:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:17:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:17:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:17:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:18:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:18:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:18:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:18:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:18:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:18:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:18:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:18:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:18:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:18:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:18:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:19:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:19:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:19:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:19:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:19:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:19:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:19:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:19:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:19:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:19:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:19:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:19:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:20:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:20:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:20:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:20:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:20:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:20:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:20:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:20:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:20:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:20:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:20:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:21:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:21:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:21:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:21:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:21:22:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-24 02:21:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:21:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:21:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:21:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:21:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:21:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:21:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:21:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:22:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:22:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:22:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:22:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:22:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:22:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:22:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:22:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:22:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:22:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:22:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:23:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:23:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:23:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:23:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:23:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:23:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:23:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:23:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:23:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:23:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:23:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:24:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:24:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:24:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:24:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:24:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:24:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:24:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:24:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:24:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:24:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:24:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:24:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:25:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:25:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:25:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:25:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:25:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:25:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:25:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:25:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:25:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:25:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:25:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:26:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:26:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:26:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:26:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:26:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:26:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:26:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:26:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:26:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:26:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:26:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:26:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:27:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:27:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:27:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:27:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:27:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:27:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:27:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:27:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:27:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:27:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:27:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:28:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:28:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:28:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:28:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:28:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:28:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 02:28:28:[cuda:0] Epoch 3000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 02:28:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 02:28:35:mr: 241.028 INFO:Trainer:2024-05-24 02:28:35:mrr: 0.312 INFO:Trainer:2024-05-24 02:28:35:hits@1: 0.231 INFO:Trainer:2024-05-24 02:28:35:hits@3: 0.34 INFO:Trainer:2024-05-24 02:28:35:hits@10: 0.473 INFO:Trainer:2024-05-24 02:28:35:mr_type: 124.4 INFO:Trainer:2024-05-24 02:28:35:mrr_type: 0.337 INFO:Trainer:2024-05-24 02:28:35:hits@1_type: 0.253 INFO:Trainer:2024-05-24 02:28:35:hits@3_type: 0.365 INFO:Trainer:2024-05-24 02:28:35:hits@10_type: 0.504 INFO:EarlyStopping:2024-05-24 02:28:35:Validation score improved (0.472000 --> 0.473000). Saving model ... INFO:Trainer:2024-05-24 02:28:35:[cuda:0] Epoch [3000/10000] | Batchsize: 1024 | loss: 0.226417 | 7.47312 seconds/epoch DEBUG:connectionpool:2024-05-24 02:28:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:28:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:28:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:28:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:28:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:29:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:29:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:29:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:29:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:29:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:29:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:29:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:29:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:29:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:29:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:29:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:30:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:30:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:30:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:30:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:30:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:30:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:30:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:30:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:30:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:30:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:30:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:30:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:31:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:31:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:31:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:31:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:31:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:31:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:31:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:31:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:31:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:31:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:31:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:32:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:32:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:32:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:32:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:32:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:32:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:32:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:32:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:32:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:32:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:32:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:33:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:33:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:33:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:33:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:33:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:33:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:33:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:33:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:33:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:33:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:33:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:33:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:34:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:34:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:34:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:34:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:34:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:34:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:34:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:34:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:34:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:34:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:34:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:35:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:35:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:35:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:35:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:35:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:35:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:35:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:35:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:35:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:35:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:35:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:35:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:36:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:36:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:36:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:36:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:36:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:36:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:36:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:36:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:36:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:36:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:36:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:37:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:37:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:37:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:37:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:37:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:37:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:37:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:37:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:37:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:37:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:37:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:37:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:38:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:38:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:38:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:38:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:38:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:38:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:38:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:38:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:38:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:38:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:38:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:39:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:39:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:39:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:39:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:39:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:39:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:39:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:39:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:39:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:39:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:39:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:40:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:40:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:40:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:40:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:40:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:40:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:40:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:40:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:40:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:40:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:40:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:40:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 02:40:58:[cuda:0] Epoch 3100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 02:41:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 02:41:04:mr: 239.935 INFO:Trainer:2024-05-24 02:41:04:mrr: 0.313 INFO:Trainer:2024-05-24 02:41:04:hits@1: 0.23 INFO:Trainer:2024-05-24 02:41:04:hits@3: 0.341 INFO:Trainer:2024-05-24 02:41:04:hits@10: 0.474 INFO:Trainer:2024-05-24 02:41:04:mr_type: 122.988 INFO:Trainer:2024-05-24 02:41:04:mrr_type: 0.338 INFO:Trainer:2024-05-24 02:41:04:hits@1_type: 0.252 INFO:Trainer:2024-05-24 02:41:04:hits@3_type: 0.366 INFO:Trainer:2024-05-24 02:41:04:hits@10_type: 0.505 INFO:EarlyStopping:2024-05-24 02:41:04:Validation score improved (0.473000 --> 0.474000). Saving model ... INFO:Trainer:2024-05-24 02:41:04:[cuda:0] Epoch [3100/10000] | Batchsize: 1024 | loss: 0.225912 | 7.47390 seconds/epoch DEBUG:connectionpool:2024-05-24 02:41:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:41:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:41:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:41:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:41:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:41:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:41:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:41:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:41:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:41:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:42:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:42:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:42:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:42:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:42:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:42:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:42:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:42:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:42:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:42:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:42:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:42:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:43:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:43:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:43:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:43:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:43:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:43:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:43:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:43:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:43:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:43:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:43:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:44:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:44:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:44:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:44:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:44:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:44:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:44:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:44:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:44:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:44:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:44:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:45:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:45:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:45:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:45:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:45:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:45:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:45:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:45:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:45:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:45:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:45:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:45:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:46:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:46:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:46:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:46:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:46:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:46:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:46:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:46:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:46:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:46:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:46:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:47:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:47:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:47:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:47:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:47:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:47:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:47:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:47:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:47:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:47:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:47:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:47:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:48:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:48:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:48:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:48:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:48:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:48:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:48:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:48:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:48:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:48:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:48:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:49:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:49:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:49:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:49:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:49:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:49:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:49:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:49:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:49:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:49:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:49:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:50:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:50:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:50:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:50:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:50:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:50:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:50:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:50:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:50:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:50:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:50:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:50:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:51:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:51:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:51:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:51:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:51:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:51:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:51:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:51:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:51:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:51:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:51:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:52:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:52:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:52:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:52:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:52:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:52:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:52:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:52:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:52:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:52:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:52:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:52:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:53:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:53:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:53:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:53:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 02:53:23:[cuda:0] Epoch 3200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 02:53:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:53:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 02:53:29:mr: 238.626 INFO:Trainer:2024-05-24 02:53:29:mrr: 0.314 INFO:Trainer:2024-05-24 02:53:29:hits@1: 0.232 INFO:Trainer:2024-05-24 02:53:29:hits@3: 0.342 INFO:Trainer:2024-05-24 02:53:29:hits@10: 0.476 INFO:Trainer:2024-05-24 02:53:29:mr_type: 121.983 INFO:Trainer:2024-05-24 02:53:29:mrr_type: 0.339 INFO:Trainer:2024-05-24 02:53:29:hits@1_type: 0.254 INFO:Trainer:2024-05-24 02:53:29:hits@3_type: 0.367 INFO:Trainer:2024-05-24 02:53:29:hits@10_type: 0.508 INFO:EarlyStopping:2024-05-24 02:53:29:Validation score improved (0.474000 --> 0.476000). Saving model ... INFO:Trainer:2024-05-24 02:53:29:[cuda:0] Epoch [3200/10000] | Batchsize: 1024 | loss: 0.225409 | 7.47304 seconds/epoch DEBUG:connectionpool:2024-05-24 02:53:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:53:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:53:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:53:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:53:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:54:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:54:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:54:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:54:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:54:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:54:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:54:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:54:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:54:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:54:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:54:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:54:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:55:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:55:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:55:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:55:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:55:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:55:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:55:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:55:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:55:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:55:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:55:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:56:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:56:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:56:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:56:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:56:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:56:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:56:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:56:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:56:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:56:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:56:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:56:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:57:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:57:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:57:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:57:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:57:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:57:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:57:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:57:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:57:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:57:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:57:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:58:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:58:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:58:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:58:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:58:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:58:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:58:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:58:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:58:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:58:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:58:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:58:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:59:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:59:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:59:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:59:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:59:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:59:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:59:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:59:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:59:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:59:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 02:59:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:00:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:00:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:00:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:00:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:00:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:00:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:00:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:00:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:00:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:00:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:00:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:01:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:01:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:01:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:01:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:01:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:01:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:01:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:01:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:01:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:01:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:01:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:01:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:02:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:02:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:02:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:02:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:02:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:02:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:02:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:02:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:02:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:02:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:02:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:03:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:03:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:03:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:03:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:03:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:03:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:03:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:03:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:03:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:03:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:03:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:03:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:04:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:04:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:04:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:04:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:04:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:04:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:04:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:04:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:04:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:04:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:04:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:05:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:05:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:05:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:05:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:05:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:05:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:05:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:05:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:05:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:05:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 03:05:50:[cuda:0] Epoch 3300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 03:05:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 03:05:56:mr: 237.589 INFO:Trainer:2024-05-24 03:05:56:mrr: 0.315 INFO:Trainer:2024-05-24 03:05:56:hits@1: 0.233 INFO:Trainer:2024-05-24 03:05:56:hits@3: 0.343 INFO:Trainer:2024-05-24 03:05:56:hits@10: 0.477 INFO:Trainer:2024-05-24 03:05:56:mr_type: 120.779 INFO:Trainer:2024-05-24 03:05:56:mrr_type: 0.34 INFO:Trainer:2024-05-24 03:05:56:hits@1_type: 0.255 INFO:Trainer:2024-05-24 03:05:56:hits@3_type: 0.368 INFO:Trainer:2024-05-24 03:05:56:hits@10_type: 0.509 INFO:EarlyStopping:2024-05-24 03:05:56:Validation score improved (0.476000 --> 0.477000). Saving model ... INFO:Trainer:2024-05-24 03:05:56:[cuda:0] Epoch [3300/10000] | Batchsize: 1024 | loss: 0.224893 | 7.47297 seconds/epoch DEBUG:connectionpool:2024-05-24 03:05:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:06:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:06:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:06:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:06:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:06:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:06:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:06:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:06:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:06:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:06:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:06:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:07:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:07:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:07:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:07:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:07:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:07:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:07:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:07:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:07:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:07:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:07:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:07:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:08:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:08:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:08:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:08:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:08:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:08:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:08:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:08:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:08:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:08:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:08:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:09:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:09:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:09:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:09:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:09:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:09:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:09:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:09:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:09:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:09:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:09:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:10:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:10:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:10:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:10:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:10:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:10:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:10:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:10:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:10:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:10:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:10:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:11:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:11:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:11:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:11:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:11:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:11:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:11:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:11:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:11:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:11:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:11:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:12:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:12:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:12:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:12:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:12:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:12:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:12:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:12:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:12:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:12:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:12:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:13:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:13:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:13:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:13:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:13:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:13:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:13:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:13:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:13:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:13:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:13:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:13:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:14:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:14:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:14:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:14:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:14:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:14:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:14:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:14:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:14:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:14:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:14:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:15:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:15:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:15:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:15:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:15:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:15:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:15:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:15:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:15:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:15:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:15:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:15:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:16:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:16:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:16:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:16:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:16:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:16:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:16:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:16:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:16:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:16:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:16:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:17:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:17:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:17:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:17:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:17:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:17:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:17:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:17:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:17:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:17:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:17:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:17:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:18:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:18:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:18:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 03:18:20:[cuda:0] Epoch 3400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 03:18:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:18:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 03:18:26:mr: 236.583 INFO:Trainer:2024-05-24 03:18:26:mrr: 0.316 INFO:Trainer:2024-05-24 03:18:26:hits@1: 0.233 INFO:Trainer:2024-05-24 03:18:26:hits@3: 0.344 INFO:Trainer:2024-05-24 03:18:26:hits@10: 0.479 INFO:Trainer:2024-05-24 03:18:26:mr_type: 120.591 INFO:Trainer:2024-05-24 03:18:26:mrr_type: 0.341 INFO:Trainer:2024-05-24 03:18:26:hits@1_type: 0.255 INFO:Trainer:2024-05-24 03:18:26:hits@3_type: 0.369 INFO:Trainer:2024-05-24 03:18:26:hits@10_type: 0.51 INFO:EarlyStopping:2024-05-24 03:18:26:Validation score improved (0.477000 --> 0.479000). Saving model ... INFO:Trainer:2024-05-24 03:18:26:[cuda:0] Epoch [3400/10000] | Batchsize: 1024 | loss: 0.224410 | 7.47387 seconds/epoch DEBUG:connectionpool:2024-05-24 03:18:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:18:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:18:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:18:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:18:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:18:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:19:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:19:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:19:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:19:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:19:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:19:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:19:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:19:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:19:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:19:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:19:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:19:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:20:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:20:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:20:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:20:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:20:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:20:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:20:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:20:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:20:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:20:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:20:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:21:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:21:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:21:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:21:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:21:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:21:28:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-24 03:21:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:21:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:21:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:21:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:21:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:21:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:22:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:22:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:22:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:22:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:22:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:22:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:22:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:22:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:22:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:22:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:22:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:22:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:23:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:23:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:23:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:23:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:23:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:23:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:23:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:23:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:23:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:23:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:23:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:24:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:24:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:24:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:24:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:24:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:24:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:24:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:24:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:24:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:24:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:24:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:24:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:25:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:25:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:25:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:25:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:25:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:25:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:25:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:25:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:25:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:25:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:25:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:26:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:26:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:26:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:26:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:26:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:26:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:26:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:26:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:26:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:26:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:26:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:26:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:27:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:27:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:27:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:27:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:27:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:27:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:27:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:27:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:27:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:27:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:27:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:28:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:28:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:28:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:28:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:28:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:28:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:28:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:28:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:28:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:28:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:28:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:28:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:29:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:29:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:29:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:29:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:29:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:29:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:29:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:29:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:29:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:29:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:29:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:30:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:30:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:30:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:30:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:30:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:30:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:30:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:30:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:30:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:30:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 03:30:50:[cuda:0] Epoch 3500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 03:30:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 03:30:55:mr: 236.689 INFO:Trainer:2024-05-24 03:30:55:mrr: 0.316 INFO:Trainer:2024-05-24 03:30:55:hits@1: 0.233 INFO:Trainer:2024-05-24 03:30:55:hits@3: 0.343 INFO:Trainer:2024-05-24 03:30:55:hits@10: 0.479 INFO:Trainer:2024-05-24 03:30:55:mr_type: 120.453 INFO:Trainer:2024-05-24 03:30:55:mrr_type: 0.341 INFO:Trainer:2024-05-24 03:30:55:hits@1_type: 0.256 INFO:Trainer:2024-05-24 03:30:55:hits@3_type: 0.368 INFO:Trainer:2024-05-24 03:30:55:hits@10_type: 0.511 INFO:EarlyStopping:2024-05-24 03:30:55:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-05-24 03:30:55:[cuda:0] Epoch [3500/10000] | Batchsize: 1024 | loss: 0.224401 | 7.47448 seconds/epoch DEBUG:connectionpool:2024-05-24 03:30:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:31:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:31:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:31:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:31:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:31:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:31:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:31:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:31:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:31:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:31:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:31:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:32:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:32:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:32:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:32:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:32:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:32:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:32:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:32:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:32:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:32:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:32:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:33:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:33:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:33:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:33:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:33:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:33:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:33:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:33:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:33:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:33:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:33:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:33:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:34:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:34:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:34:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:34:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:34:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:34:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:34:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:34:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:34:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:34:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:34:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:35:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:35:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:35:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:35:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:35:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:35:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:35:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:35:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:35:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:35:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:35:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:35:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:36:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:36:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:36:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:36:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:36:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:36:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:36:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:36:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:36:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:36:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:36:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:37:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:37:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:37:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:37:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:37:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:37:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:37:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:37:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:37:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:37:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:37:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:38:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:38:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:38:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:38:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:38:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:38:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:38:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:38:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:38:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:38:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:38:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:38:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:39:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:39:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:39:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:39:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:39:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:39:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:39:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:39:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:39:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:39:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:39:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:40:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:40:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:40:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:40:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:40:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:40:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:40:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:40:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:40:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:40:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:40:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:40:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:41:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:41:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:41:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:41:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:41:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:41:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:41:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:41:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:41:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:41:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:41:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:42:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:42:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:42:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:42:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:42:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:42:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:42:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:42:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:42:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:42:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:42:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:42:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:43:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:43:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:43:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 03:43:19:[cuda:0] Epoch 3600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 03:43:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 03:43:25:mr: 236.211 INFO:Trainer:2024-05-24 03:43:25:mrr: 0.316 INFO:Trainer:2024-05-24 03:43:25:hits@1: 0.233 INFO:Trainer:2024-05-24 03:43:25:hits@3: 0.344 INFO:Trainer:2024-05-24 03:43:25:hits@10: 0.479 INFO:Trainer:2024-05-24 03:43:25:mr_type: 120.347 INFO:Trainer:2024-05-24 03:43:25:mrr_type: 0.341 INFO:Trainer:2024-05-24 03:43:25:hits@1_type: 0.255 INFO:Trainer:2024-05-24 03:43:25:hits@3_type: 0.368 INFO:Trainer:2024-05-24 03:43:25:hits@10_type: 0.511 INFO:EarlyStopping:2024-05-24 03:43:25:EarlyStopping counter: 2 / 2 INFO:Trainer:2024-05-24 03:43:25:[cuda:0] Send an early stopping signal INFO:Trainer:2024-05-24 03:43:25:[cuda:0] The model training is completed, taking a total of 26910.28981 seconds. INFO:Trainer:2024-05-24 03:43:25:[cuda:0] Model saved at ./model.pth. INFO:Trainer:2024-05-24 03:43:25:[cuda:0] The model starts evaluating in the test set. DEBUG:connectionpool:2024-05-24 03:43:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:43:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 03:43:32:mr: 238.987 INFO:Trainer:2024-05-24 03:43:32:mrr: 0.314 INFO:Trainer:2024-05-24 03:43:32:hits@1: 0.23 INFO:Trainer:2024-05-24 03:43:32:hits@3: 0.343 INFO:Trainer:2024-05-24 03:43:32:hits@10: 0.478 INFO:Trainer:2024-05-24 03:43:32:mr_type: 121.411 INFO:Trainer:2024-05-24 03:43:32:mrr_type: 0.338 INFO:Trainer:2024-05-24 03:43:32:hits@1_type: 0.252 INFO:Trainer:2024-05-24 03:43:32:hits@3_type: 0.366 INFO:Trainer:2024-05-24 03:43:32:hits@10_type: 0.507 wandb: - 0.014 MB of 0.014 MB uploaded wandb: \ 0.014 MB of 0.569 MB uploaded wandb: | 0.531 MB of 0.569 MB uploaded DEBUG:connectionpool:2024-05-24 03:43:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: / 0.567 MB of 0.569 MB uploaded wandb: - 0.567 MB of 0.569 MB uploaded wandb: \ 0.567 MB of 0.569 MB uploaded wandb: | 0.569 MB of 0.569 MB uploaded wandb: wandb: wandb: Run history: wandb: duration โ– wandb: test/hits@1 โ– wandb: test/hits@10 โ– wandb: test/hits@10_type โ– wandb: test/hits@1_type โ– wandb: test/hits@3 โ– wandb: test/hits@3_type โ– wandb: test/mr โ– wandb: test/mr_type โ– wandb: test/mrr โ– wandb: test/mrr_type โ– wandb: train/epoch โ–โ–โ–โ–‚โ–‚โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆ wandb: train/train_loss โ–ˆโ–†โ–…โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ– wandb: val/epoch โ–โ–โ–โ–‚โ–‚โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆ wandb: val/hits@1 โ–โ–ƒโ–„โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@10 โ–โ–ƒโ–„โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@10_type โ–โ–‚โ–‚โ–ƒโ–ƒโ–„โ–„โ–…โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@1_type โ–โ–‚โ–‚โ–ƒโ–ƒโ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@3 โ–โ–ƒโ–„โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@3_type โ–โ–‚โ–‚โ–ƒโ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/mr โ–ˆโ–…โ–ƒโ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ– wandb: val/mr_type โ–ˆโ–ˆโ–‡โ–†โ–…โ–…โ–„โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ– wandb: val/mrr โ–โ–ƒโ–„โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/mrr_type โ–โ–‚โ–‚โ–ƒโ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: wandb: Run summary: wandb: duration 26910.28981 wandb: test/hits@1 0.23 wandb: test/hits@10 0.478 wandb: test/hits@10_type 0.507 wandb: test/hits@1_type 0.252 wandb: test/hits@3 0.343 wandb: test/hits@3_type 0.366 wandb: test/mr 238.987 wandb: test/mr_type 121.411 wandb: test/mrr 0.314 wandb: test/mrr_type 0.338 wandb: train/epoch 3500 wandb: train/train_loss 0.2244 wandb: val/epoch 3599 wandb: val/hits@1 0.233 wandb: val/hits@10 0.479 wandb: val/hits@10_type 0.511 wandb: val/hits@1_type 0.255 wandb: val/hits@3 0.344 wandb: val/hits@3_type 0.368 wandb: val/mr 236.211 wandb: val/mr_type 120.347 wandb: val/mrr 0.316 wandb: val/mrr_type 0.341 wandb: wandb: ๐Ÿš€ View run worldly-sweep-2 at: https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/6bgcsi0p wandb: โญ๏ธ View project at: https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) wandb: Find logs at: ./wandb/run-20240523_201441-6bgcsi0p/logs DEBUG:connectionpool:2024-05-24 03:43:39:Resetting dropped connection: o151352.ingest.sentry.io DEBUG:connectionpool:2024-05-24 03:43:40:https://o151352.ingest.sentry.io:443 "POST /api/4504800232407040/envelope/ HTTP/1.1" 200 0 DEBUG:connectionpool:2024-05-24 03:43:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: Agent Starting Run: wsmfz2ez with config: wandb: batch_size: 1024 wandb: dataloader: KGEDataLoader wandb: decoder_model: ConvE wandb: delta: 0.0001 wandb: device: cuda:0 wandb: dim: 100 wandb: ent_file: entity2id.txt wandb: epochs: 10000 wandb: fet_drop: 0.2 wandb: hid_drop: 0.3 wandb: in_path: ../../benchmarks/FB15K237/ wandb: log_interval: 100 wandb: loss: CompGCNLoss wandb: lr: 0.0034700606547572476 wandb: margin: 40 wandb: metric: hits@10 wandb: model: CompGCN wandb: neg_ent: 1 wandb: num_workers: 16 wandb: opn: mult wandb: opt_method: sgd wandb: patience: 2 wandb: prediction: tail wandb: rel_file: relation2id.txt wandb: save_path: ./model.pth wandb: smoothing: 0.1 wandb: strategy: CompGCNSampling wandb: test_batch_size: 256 wandb: test_file: test2id.txt wandb: test_sampler: CompGCNTestSampler wandb: tester: Tester wandb: train_file: train2id.txt wandb: train_sampler: CompGCNSampler wandb: trainer: Trainer wandb: type_constrain: True wandb: use_early_stopping: True wandb: use_gpu: True wandb: use_tqdm: False wandb: valid_file: valid2id.txt wandb: valid_interval: 100 DEBUG:cmd:2024-05-24 03:43:43:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:cmd:2024-05-24 03:43:43:Popen(['git', 'rev-parse', '--show-toplevel'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=None, shell=False, universal_newlines=False) DEBUG:connectionpool:2024-05-24 03:43:43:Starting new HTTPS connection (1): api.wandb.ai:443 DEBUG:connectionpool:2024-05-24 03:43:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:43:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:cmd:2024-05-24 03:43:44:Popen(['git', 'cat-file', '--batch-check'], cwd=/home/luyanfeng/my_code/github/pybind11-OpenKE, stdin=, shell=False, universal_newlines=False) wandb: wandb version 0.17.0 is available! To upgrade, please run: wandb: $ pip install wandb --upgrade wandb: Tracking run with wandb version 0.16.6 wandb: Run data is saved locally in /home/luyanfeng/my_code/github/pybind11-OpenKE/examples/CompGCN/wandb/run-20240524_034344-wsmfz2ez wandb: Run `wandb offline` to turn off syncing. wandb: Syncing run glamorous-sweep-3 wandb: โญ๏ธ View project at https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: ๐Ÿงน View sweep at https://wandb.ai/3555028709/pybind11-ke-sweeps/sweeps/fnx5v46t wandb: ๐Ÿš€ View run at https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/wsmfz2ez DEBUG:connectionpool:2024-05-24 03:43:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 03:43:51:[cuda:0] Initialization completed, start model training. DEBUG:connectionpool:2024-05-24 03:43:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:43:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:44:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:44:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:44:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:44:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:44:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:44:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:44:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:44:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:44:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:44:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:44:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:45:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:45:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:45:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:45:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:45:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:45:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:45:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:45:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:45:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:45:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:45:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:45:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:46:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:46:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:46:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:46:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:46:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:46:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:46:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:46:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:46:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:46:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:46:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:47:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:47:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:47:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:47:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:47:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:47:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:47:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:47:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:47:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:47:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:47:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:48:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:48:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:48:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:48:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:48:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:48:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:48:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:48:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:48:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:48:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:48:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:48:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:49:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:49:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:49:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:49:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:49:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:49:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:49:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:49:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:49:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:49:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:49:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:50:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:50:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:50:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:50:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:50:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:50:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:50:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:50:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:50:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:50:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:50:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:50:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:51:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:51:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:51:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:51:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:51:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:51:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:51:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:51:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:51:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:51:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:51:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:52:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:52:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:52:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:52:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:52:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:52:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:52:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:52:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:52:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:52:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:52:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:53:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:53:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:53:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:53:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:53:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:53:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:53:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:53:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:53:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:53:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:53:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:53:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:54:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:54:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:54:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:54:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:54:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:54:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:54:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:54:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:54:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:54:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:54:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:55:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:55:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:55:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:55:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:55:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:55:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:55:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:55:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:55:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:55:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:55:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:55:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:56:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:56:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 03:56:10:[cuda:0] Epoch 100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 03:56:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 03:56:16:mr: 6339.619 INFO:Trainer:2024-05-24 03:56:16:mrr: 0.026 INFO:Trainer:2024-05-24 03:56:16:hits@1: 0.021 INFO:Trainer:2024-05-24 03:56:16:hits@3: 0.027 INFO:Trainer:2024-05-24 03:56:16:hits@10: 0.035 INFO:Trainer:2024-05-24 03:56:16:mr_type: 376.556 INFO:Trainer:2024-05-24 03:56:16:mrr_type: 0.132 INFO:Trainer:2024-05-24 03:56:16:hits@1_type: 0.094 INFO:Trainer:2024-05-24 03:56:16:hits@3_type: 0.128 INFO:Trainer:2024-05-24 03:56:16:hits@10_type: 0.198 INFO:EarlyStopping:2024-05-24 03:56:16:Validation score improved (-inf --> 0.035000). Saving model ... INFO:Trainer:2024-05-24 03:56:16:[cuda:0] Epoch [ 100/10000] | Batchsize: 1024 | loss: 0.433294 | 7.39597 seconds/epoch DEBUG:connectionpool:2024-05-24 03:56:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:56:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:56:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:56:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:56:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:56:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:56:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:56:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:57:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:57:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:57:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:57:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:57:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:57:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:57:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:57:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:57:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:57:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:57:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:57:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:58:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:58:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:58:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:58:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:58:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:58:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:58:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:58:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:58:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:58:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:58:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:59:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:59:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:59:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:59:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:59:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:59:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:59:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:59:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:59:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:59:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 03:59:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:00:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:00:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:00:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:00:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:00:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:00:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:00:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:00:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:00:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:00:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:00:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:00:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:01:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:01:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:01:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:02:19:Starting new HTTPS connection (3): api.wandb.ai:443 wandb: Network error (ConnectTimeout), entering retry loop. DEBUG:connectionpool:2024-05-24 04:03:21:Starting new HTTPS connection (4): api.wandb.ai:443 wandb: Network error (ConnectTimeout), entering retry loop. DEBUG:connectionpool:2024-05-24 04:04:27:Starting new HTTPS connection (5): api.wandb.ai:443 DEBUG:connectionpool:2024-05-24 04:05:36:Starting new HTTPS connection (6): api.wandb.ai:443 DEBUG:connectionpool:2024-05-24 04:06:54:Starting new HTTPS connection (7): api.wandb.ai:443 DEBUG:connectionpool:2024-05-24 04:08:28:Starting new HTTPS connection (8): api.wandb.ai:443 INFO:Trainer:2024-05-24 04:08:30:[cuda:0] Epoch 200 | The model starts evaluation on the validation set. INFO:Trainer:2024-05-24 04:08:36:mr: 5938.341 INFO:Trainer:2024-05-24 04:08:36:mrr: 0.06 INFO:Trainer:2024-05-24 04:08:36:hits@1: 0.056 INFO:Trainer:2024-05-24 04:08:36:hits@3: 0.061 INFO:Trainer:2024-05-24 04:08:36:hits@10: 0.065 INFO:Trainer:2024-05-24 04:08:36:mr_type: 372.332 INFO:Trainer:2024-05-24 04:08:36:mrr_type: 0.15 INFO:Trainer:2024-05-24 04:08:36:hits@1_type: 0.111 INFO:Trainer:2024-05-24 04:08:36:hits@3_type: 0.148 INFO:Trainer:2024-05-24 04:08:36:hits@10_type: 0.22 INFO:EarlyStopping:2024-05-24 04:08:36:Validation score improved (0.035000 --> 0.065000). Saving model ... INFO:Trainer:2024-05-24 04:08:36:[cuda:0] Epoch [ 200/10000] | Batchsize: 1024 | loss: 0.405261 | 7.39519 seconds/epoch DEBUG:connectionpool:2024-05-24 04:10:36:Starting new HTTPS connection (9): api.wandb.ai:443 DEBUG:connectionpool:2024-05-24 04:14:15:Starting new HTTPS connection (10): api.wandb.ai:443 DEBUG:connectionpool:2024-05-24 04:19:38:Starting new HTTPS connection (11): api.wandb.ai:443 DEBUG:connectionpool:2024-05-24 04:20:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:20:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:20:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:20:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:20:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:20:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:20:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 04:20:51:[cuda:0] Epoch 300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 04:20:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:20:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 04:20:57:mr: 5656.594 INFO:Trainer:2024-05-24 04:20:57:mrr: 0.069 INFO:Trainer:2024-05-24 04:20:57:hits@1: 0.062 INFO:Trainer:2024-05-24 04:20:57:hits@3: 0.071 INFO:Trainer:2024-05-24 04:20:57:hits@10: 0.079 INFO:Trainer:2024-05-24 04:20:57:mr_type: 368.694 INFO:Trainer:2024-05-24 04:20:57:mrr_type: 0.16 INFO:Trainer:2024-05-24 04:20:57:hits@1_type: 0.118 INFO:Trainer:2024-05-24 04:20:57:hits@3_type: 0.161 INFO:Trainer:2024-05-24 04:20:57:hits@10_type: 0.24 INFO:EarlyStopping:2024-05-24 04:20:57:Validation score improved (0.065000 --> 0.079000). Saving model ... INFO:Trainer:2024-05-24 04:20:58:[cuda:0] Epoch [ 300/10000] | Batchsize: 1024 | loss: 0.388531 | 7.40224 seconds/epoch DEBUG:connectionpool:2024-05-24 04:21:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:21:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:21:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:21:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:21:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:21:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:21:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:21:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:21:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:21:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:21:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:22:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:22:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:22:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:22:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:22:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:22:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:22:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:22:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:22:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:22:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:22:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:22:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:23:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:23:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:23:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:23:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:23:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:23:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:23:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:23:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:23:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:23:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:23:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:24:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:24:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:24:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:24:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:24:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:24:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:24:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:24:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:24:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:24:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:24:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:25:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:25:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:25:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:25:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:25:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:25:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:25:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:25:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:25:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:25:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:25:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:25:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:26:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:26:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:26:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:26:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:26:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:26:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:26:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:26:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:26:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:26:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:26:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:27:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:27:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:27:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:27:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:27:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:27:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:27:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:27:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:27:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:27:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:27:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:27:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:28:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:28:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:28:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:28:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:28:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:28:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:28:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:28:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:28:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:28:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:28:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:29:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:29:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:29:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:29:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:29:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:29:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:29:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:29:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:29:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:29:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:29:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:29:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:30:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:30:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:30:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:30:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:30:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:30:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:30:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:30:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:30:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:30:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:30:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:31:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:31:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:31:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:31:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:31:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:31:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:31:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:31:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:31:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:31:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:31:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:32:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:32:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:32:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:32:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:32:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:32:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:32:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:32:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:32:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:32:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:32:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:32:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:33:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:33:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:33:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:33:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 04:33:19:[cuda:0] Epoch 400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 04:33:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 04:33:25:mr: 5429.568 INFO:Trainer:2024-05-24 04:33:25:mrr: 0.075 INFO:Trainer:2024-05-24 04:33:25:hits@1: 0.066 INFO:Trainer:2024-05-24 04:33:25:hits@3: 0.078 INFO:Trainer:2024-05-24 04:33:25:hits@10: 0.09 INFO:Trainer:2024-05-24 04:33:25:mr_type: 365.96 INFO:Trainer:2024-05-24 04:33:25:mrr_type: 0.17 INFO:Trainer:2024-05-24 04:33:25:hits@1_type: 0.125 INFO:Trainer:2024-05-24 04:33:25:hits@3_type: 0.175 INFO:Trainer:2024-05-24 04:33:25:hits@10_type: 0.254 INFO:EarlyStopping:2024-05-24 04:33:25:Validation score improved (0.079000 --> 0.090000). Saving model ... INFO:Trainer:2024-05-24 04:33:25:[cuda:0] Epoch [ 400/10000] | Batchsize: 1024 | loss: 0.378169 | 7.42080 seconds/epoch DEBUG:connectionpool:2024-05-24 04:33:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:33:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:33:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:33:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:33:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:33:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:34:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:34:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:34:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:34:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:34:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:34:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:34:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:34:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:34:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:34:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:34:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:34:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:35:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:35:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:35:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:35:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:35:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:35:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:35:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:35:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:35:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:35:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:35:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:36:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:36:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:36:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:36:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:36:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:36:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:36:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:36:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:36:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:36:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:36:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:37:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:37:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:37:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:37:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:37:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:37:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:37:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:37:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:37:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:37:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:37:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:37:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:38:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:38:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:38:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:38:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:38:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:38:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:38:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:38:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:38:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:38:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:38:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:39:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:39:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:39:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:39:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:39:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:39:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:39:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:39:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:39:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:39:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:39:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:39:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:40:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:40:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:40:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:40:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:40:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:40:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:40:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:40:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:40:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:40:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:40:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:41:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:41:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:41:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:41:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:41:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:41:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:41:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:41:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:41:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:41:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:41:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:41:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:42:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:42:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:42:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:42:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:42:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:42:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:42:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:42:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:42:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:42:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:42:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:43:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:43:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:43:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:43:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:43:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:43:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:43:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:43:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:43:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:43:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:43:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:44:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:44:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:44:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:44:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:44:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:44:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:44:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:44:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:44:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:44:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:44:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:44:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:45:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:45:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:45:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:45:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:45:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:45:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:45:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:45:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 04:45:43:[cuda:0] Epoch 500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 04:45:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 04:45:49:mr: 5228.745 INFO:Trainer:2024-05-24 04:45:49:mrr: 0.081 INFO:Trainer:2024-05-24 04:45:49:hits@1: 0.07 INFO:Trainer:2024-05-24 04:45:49:hits@3: 0.082 INFO:Trainer:2024-05-24 04:45:49:hits@10: 0.103 INFO:Trainer:2024-05-24 04:45:49:mr_type: 362.877 INFO:Trainer:2024-05-24 04:45:49:mrr_type: 0.178 INFO:Trainer:2024-05-24 04:45:49:hits@1_type: 0.13 INFO:Trainer:2024-05-24 04:45:49:hits@3_type: 0.186 INFO:Trainer:2024-05-24 04:45:49:hits@10_type: 0.267 INFO:EarlyStopping:2024-05-24 04:45:49:Validation score improved (0.090000 --> 0.103000). Saving model ... INFO:Trainer:2024-05-24 04:45:49:[cuda:0] Epoch [ 500/10000] | Batchsize: 1024 | loss: 0.371111 | 7.42477 seconds/epoch DEBUG:connectionpool:2024-05-24 04:45:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:45:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:46:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:46:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:46:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:46:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:46:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:46:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:46:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:46:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:46:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:46:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:46:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:46:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:47:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:47:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:47:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:47:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:47:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:47:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:47:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:47:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:47:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:47:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:47:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:48:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:48:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:48:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:48:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:48:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:48:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:48:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:48:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:48:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:48:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:48:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:48:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:49:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:49:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:49:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:49:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:49:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:49:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:49:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:49:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:49:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:49:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:49:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:50:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:50:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:50:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:50:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:50:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:50:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:50:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:50:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:50:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:50:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:50:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:50:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:51:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:51:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:51:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:51:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:51:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:51:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:51:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:51:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:51:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:51:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:51:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:52:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:52:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:52:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:52:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:52:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:52:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:52:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:52:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:52:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:52:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:52:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:53:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:53:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:53:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:53:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:53:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:53:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:53:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:53:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:53:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:53:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:53:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:53:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:54:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:54:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:54:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:54:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:54:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:54:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:54:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:54:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:54:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:54:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:54:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:55:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:55:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:55:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:55:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:55:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:55:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:55:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:55:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:55:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:55:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:55:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:55:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:56:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:56:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:56:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:56:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:56:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:56:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:56:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:56:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:56:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:56:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:56:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:57:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:57:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:57:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:57:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:57:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:57:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:57:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:57:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:57:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:57:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:57:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:58:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:58:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 04:58:10:[cuda:0] Epoch 600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 04:58:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:58:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 04:58:16:mr: 5047.49 INFO:Trainer:2024-05-24 04:58:16:mrr: 0.087 INFO:Trainer:2024-05-24 04:58:16:hits@1: 0.074 INFO:Trainer:2024-05-24 04:58:16:hits@3: 0.086 INFO:Trainer:2024-05-24 04:58:16:hits@10: 0.115 INFO:Trainer:2024-05-24 04:58:16:mr_type: 360.498 INFO:Trainer:2024-05-24 04:58:16:mrr_type: 0.185 INFO:Trainer:2024-05-24 04:58:16:hits@1_type: 0.135 INFO:Trainer:2024-05-24 04:58:16:hits@3_type: 0.194 INFO:Trainer:2024-05-24 04:58:16:hits@10_type: 0.277 INFO:EarlyStopping:2024-05-24 04:58:16:Validation score improved (0.103000 --> 0.115000). Saving model ... INFO:Trainer:2024-05-24 04:58:16:[cuda:0] Epoch [ 600/10000] | Batchsize: 1024 | loss: 0.366524 | 7.43209 seconds/epoch DEBUG:connectionpool:2024-05-24 04:58:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:58:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:58:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:58:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:58:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:58:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:58:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:58:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:59:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:59:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:59:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:59:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:59:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:59:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:59:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:59:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:59:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:59:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 04:59:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:00:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:00:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:00:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:00:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:00:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:00:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:00:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:00:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:00:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:00:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:00:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:00:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:01:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:01:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:01:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:01:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:01:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:01:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:01:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:01:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:01:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:01:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:01:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:02:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:02:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:02:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:02:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:02:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:02:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:02:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:02:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:02:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:02:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:02:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:03:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:03:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:03:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:03:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:03:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:03:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:03:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:03:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:03:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:03:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:03:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:03:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:04:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:04:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:04:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:04:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:04:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:04:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:04:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:04:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:04:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:04:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:04:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:05:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:05:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:05:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:05:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:05:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:05:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:05:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:05:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:05:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:05:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:05:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:05:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:06:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:06:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:06:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:06:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:06:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:06:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:06:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:06:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:06:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:06:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:06:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:07:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:07:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:07:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:07:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:07:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:07:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:07:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:07:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:07:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:07:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:07:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:07:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:08:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:08:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:08:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:08:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:08:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:08:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:08:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:08:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:08:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:08:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:08:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:09:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:09:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:09:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:09:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:09:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:09:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:09:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:09:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:09:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:09:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:09:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:09:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:10:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:10:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:10:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:10:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:10:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:10:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 05:10:34:[cuda:0] Epoch 700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 05:10:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 05:10:40:mr: 4890.734 INFO:Trainer:2024-05-24 05:10:40:mrr: 0.091 INFO:Trainer:2024-05-24 05:10:40:hits@1: 0.076 INFO:Trainer:2024-05-24 05:10:40:hits@3: 0.09 INFO:Trainer:2024-05-24 05:10:40:hits@10: 0.126 INFO:Trainer:2024-05-24 05:10:40:mr_type: 358.389 INFO:Trainer:2024-05-24 05:10:40:mrr_type: 0.19 INFO:Trainer:2024-05-24 05:10:40:hits@1_type: 0.139 INFO:Trainer:2024-05-24 05:10:40:hits@3_type: 0.199 INFO:Trainer:2024-05-24 05:10:40:hits@10_type: 0.286 INFO:EarlyStopping:2024-05-24 05:10:40:Validation score improved (0.115000 --> 0.126000). Saving model ... INFO:Trainer:2024-05-24 05:10:41:[cuda:0] Epoch [ 700/10000] | Batchsize: 1024 | loss: 0.363123 | 7.43383 seconds/epoch DEBUG:connectionpool:2024-05-24 05:10:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:10:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:10:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:10:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:11:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:11:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:11:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:11:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:11:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:11:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:11:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:11:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:11:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:11:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:11:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:12:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:12:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:12:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:12:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:12:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:12:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:12:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:12:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:12:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:12:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:12:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:12:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:13:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:13:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:13:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:13:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:13:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:13:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:13:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:13:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:13:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:13:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:13:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:14:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:14:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:14:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:14:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:14:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:14:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:14:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:14:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:14:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:14:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:14:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:14:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:15:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:15:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:15:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:15:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:15:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:15:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:15:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:15:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:15:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:15:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:15:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:16:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:16:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:16:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:16:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:16:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:16:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:16:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:16:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:16:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:16:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:16:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:17:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:17:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:17:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:17:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:17:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:17:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:17:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:17:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:17:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:17:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:17:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:17:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:18:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:18:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:18:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:18:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:18:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:18:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:18:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:18:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:18:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:18:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:18:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:19:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:19:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:19:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:19:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:19:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:19:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:19:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:19:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:19:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:19:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:19:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:19:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:20:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:20:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:20:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:20:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:20:24:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-24 05:20:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:20:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:20:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:20:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:20:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:20:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:20:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:21:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:21:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:21:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:21:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:21:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:21:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:21:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:21:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:21:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:21:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:21:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:21:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:22:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:22:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:22:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:22:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:22:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:22:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:22:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:22:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:22:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:22:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:22:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:23:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 05:23:05:[cuda:0] Epoch 800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 05:23:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 05:23:11:mr: 4742.536 INFO:Trainer:2024-05-24 05:23:11:mrr: 0.096 INFO:Trainer:2024-05-24 05:23:11:hits@1: 0.077 INFO:Trainer:2024-05-24 05:23:11:hits@3: 0.099 INFO:Trainer:2024-05-24 05:23:11:hits@10: 0.135 INFO:Trainer:2024-05-24 05:23:11:mr_type: 356.371 INFO:Trainer:2024-05-24 05:23:11:mrr_type: 0.195 INFO:Trainer:2024-05-24 05:23:11:hits@1_type: 0.143 INFO:Trainer:2024-05-24 05:23:11:hits@3_type: 0.204 INFO:Trainer:2024-05-24 05:23:11:hits@10_type: 0.295 INFO:EarlyStopping:2024-05-24 05:23:11:Validation score improved (0.126000 --> 0.135000). Saving model ... INFO:Trainer:2024-05-24 05:23:12:[cuda:0] Epoch [ 800/10000] | Batchsize: 1024 | loss: 0.359621 | 7.44320 seconds/epoch DEBUG:connectionpool:2024-05-24 05:23:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:23:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:23:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:23:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:23:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:23:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:23:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:23:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:23:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:23:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:24:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:24:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:24:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:24:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:24:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:24:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:24:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:24:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:24:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:24:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:24:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:25:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:25:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:25:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:25:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:25:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:25:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:25:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:25:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:25:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:25:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:25:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:26:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:26:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:26:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:26:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:26:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:26:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:26:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:26:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:26:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:26:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:26:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:26:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:27:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:27:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:27:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:27:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:27:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:27:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:27:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:27:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:27:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:27:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:27:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:28:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:28:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:28:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:28:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:28:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:28:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:28:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:28:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:28:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:28:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:28:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:28:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:29:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:29:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:29:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:29:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:29:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:29:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:29:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:29:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:29:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:29:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:29:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:30:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:30:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:30:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:30:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:30:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:30:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:30:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:30:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:30:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:30:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:30:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:30:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:31:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:31:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:31:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:31:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:31:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:31:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:31:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:31:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:31:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:31:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:31:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:32:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:32:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:32:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:32:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:32:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:32:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:32:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:32:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:32:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:32:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:32:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:33:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:33:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:33:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:33:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:33:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:33:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:33:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:33:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:33:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:33:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:33:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:33:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:34:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:34:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:34:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:34:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:34:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:34:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:34:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:34:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:34:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:34:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:34:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:35:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:35:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:35:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:35:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:35:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:35:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:35:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 05:35:33:[cuda:0] Epoch 900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 05:35:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 05:35:39:mr: 4621.712 INFO:Trainer:2024-05-24 05:35:39:mrr: 0.103 INFO:Trainer:2024-05-24 05:35:39:hits@1: 0.083 INFO:Trainer:2024-05-24 05:35:39:hits@3: 0.108 INFO:Trainer:2024-05-24 05:35:39:hits@10: 0.145 INFO:Trainer:2024-05-24 05:35:39:mr_type: 355.279 INFO:Trainer:2024-05-24 05:35:39:mrr_type: 0.199 INFO:Trainer:2024-05-24 05:35:39:hits@1_type: 0.146 INFO:Trainer:2024-05-24 05:35:39:hits@3_type: 0.208 INFO:Trainer:2024-05-24 05:35:39:hits@10_type: 0.3 INFO:EarlyStopping:2024-05-24 05:35:39:Validation score improved (0.135000 --> 0.145000). Saving model ... INFO:Trainer:2024-05-24 05:35:40:[cuda:0] Epoch [ 900/10000] | Batchsize: 1024 | loss: 0.356849 | 7.44748 seconds/epoch DEBUG:connectionpool:2024-05-24 05:35:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:35:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:35:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:35:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:36:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:36:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:36:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:36:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:36:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:36:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:36:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:36:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:36:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:36:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:36:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:37:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:37:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:37:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:37:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:37:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:37:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:37:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:37:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:37:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:37:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:37:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:38:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:38:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:38:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:38:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:38:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:38:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:38:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:38:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:38:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:38:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:38:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:38:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:39:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:39:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:39:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:39:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:39:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:39:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:39:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:39:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:39:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:39:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:39:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:40:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:40:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:40:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:40:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:40:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:40:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:40:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:40:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:40:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:40:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:40:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:40:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:41:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:41:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:41:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:41:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:41:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:41:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:41:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:41:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:41:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:41:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:41:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:42:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:42:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:42:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:42:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:42:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:42:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:42:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:42:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:42:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:42:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:42:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:43:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:43:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:43:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:43:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:43:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:43:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:43:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:43:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:43:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:43:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:43:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:43:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:44:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:44:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:44:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:44:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:44:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:44:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:44:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:44:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:44:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:44:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:44:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:45:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:45:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:45:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:45:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:45:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:45:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:45:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:45:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:45:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:45:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:45:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:45:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:46:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:46:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:46:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:46:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:46:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:46:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:46:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:46:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:46:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:46:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:46:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:47:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:47:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:47:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:47:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:47:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:47:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:47:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:47:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:47:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:47:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:47:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 05:47:56:[cuda:0] Epoch 1000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 05:47:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 05:48:02:mr: 4512.503 INFO:Trainer:2024-05-24 05:48:02:mrr: 0.11 INFO:Trainer:2024-05-24 05:48:02:hits@1: 0.089 INFO:Trainer:2024-05-24 05:48:02:hits@3: 0.116 INFO:Trainer:2024-05-24 05:48:02:hits@10: 0.151 INFO:Trainer:2024-05-24 05:48:02:mr_type: 354.259 INFO:Trainer:2024-05-24 05:48:02:mrr_type: 0.202 INFO:Trainer:2024-05-24 05:48:02:hits@1_type: 0.148 INFO:Trainer:2024-05-24 05:48:02:hits@3_type: 0.213 INFO:Trainer:2024-05-24 05:48:02:hits@10_type: 0.305 INFO:EarlyStopping:2024-05-24 05:48:02:Validation score improved (0.145000 --> 0.151000). Saving model ... INFO:Trainer:2024-05-24 05:48:02:[cuda:0] Epoch [1000/10000] | Batchsize: 1024 | loss: 0.354387 | 7.44576 seconds/epoch DEBUG:connectionpool:2024-05-24 05:48:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:48:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:48:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:48:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:48:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:48:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:48:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:48:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:48:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:48:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:48:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:49:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:49:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:49:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:49:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:49:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:49:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:49:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:49:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:49:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:49:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:49:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:50:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:50:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:50:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:50:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:50:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:50:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:50:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:50:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:50:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:50:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:50:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:50:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:51:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:51:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:51:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:51:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:51:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:51:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:51:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:51:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:51:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:51:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:51:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:52:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:52:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:52:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:52:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:52:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:52:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:52:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:52:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:52:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:52:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:52:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:53:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:53:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:53:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:53:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:53:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:53:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:53:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:53:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:53:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:53:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:53:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:53:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:54:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:54:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:54:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:54:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:54:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:54:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:54:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:54:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:54:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:54:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:54:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:55:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:55:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:55:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:55:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:55:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:55:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:55:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:55:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:55:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:55:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:55:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:55:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:56:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:56:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:56:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:56:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:56:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:56:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:56:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:56:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:56:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:56:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:56:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:57:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:57:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:57:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:57:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:57:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:57:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:57:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:57:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:57:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:57:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:57:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:58:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:58:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:58:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:58:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:58:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:58:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:58:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:58:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:58:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:58:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:58:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:58:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:59:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:59:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:59:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:59:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:59:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:59:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:59:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:59:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:59:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:59:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 05:59:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:00:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:00:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:00:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:00:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:00:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:00:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:00:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:00:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:00:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 06:00:45:[cuda:0] Epoch 1100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 06:00:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 06:00:52:mr: 4410.725 INFO:Trainer:2024-05-24 06:00:52:mrr: 0.114 INFO:Trainer:2024-05-24 06:00:52:hits@1: 0.091 INFO:Trainer:2024-05-24 06:00:52:hits@3: 0.121 INFO:Trainer:2024-05-24 06:00:52:hits@10: 0.158 INFO:Trainer:2024-05-24 06:00:52:mr_type: 352.156 INFO:Trainer:2024-05-24 06:00:52:mrr_type: 0.205 INFO:Trainer:2024-05-24 06:00:52:hits@1_type: 0.151 INFO:Trainer:2024-05-24 06:00:52:hits@3_type: 0.217 INFO:Trainer:2024-05-24 06:00:52:hits@10_type: 0.309 INFO:EarlyStopping:2024-05-24 06:00:52:Validation score improved (0.151000 --> 0.158000). Saving model ... INFO:Trainer:2024-05-24 06:00:52:[cuda:0] Epoch [1100/10000] | Batchsize: 1024 | loss: 0.352236 | 7.46778 seconds/epoch DEBUG:connectionpool:2024-05-24 06:00:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:00:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:01:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:01:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:01:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:01:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:01:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:01:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:01:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:01:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:01:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:01:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:01:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:02:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:02:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:02:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:02:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:02:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:02:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:02:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:02:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:02:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:02:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:02:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:03:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:03:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:03:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:03:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:03:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:03:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:03:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:03:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:03:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:03:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:03:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:04:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:04:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:04:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:04:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:04:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:04:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:04:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:04:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:04:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:04:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:04:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:04:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:05:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:05:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:05:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:05:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:05:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:05:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:05:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:05:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:05:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:05:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:05:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:06:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:06:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:06:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:06:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:06:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:06:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:06:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:06:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:06:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:06:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:06:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:07:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:07:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:07:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:07:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:07:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:07:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:07:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:07:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:07:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:07:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:07:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:07:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:08:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:08:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:08:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:08:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:08:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:08:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:08:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:08:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:08:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:08:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:08:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:09:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:09:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:09:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:09:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:09:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:09:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:09:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:09:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:09:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:09:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:09:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:09:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:10:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:10:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:10:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:10:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:10:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:10:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:10:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:10:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:10:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:10:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:10:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:11:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:11:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:11:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:11:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:11:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:11:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:11:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:11:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:11:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:11:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:11:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:11:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:12:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:12:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:12:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:12:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:12:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:12:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:12:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:12:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:12:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:12:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:12:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:13:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:13:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:13:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:13:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:13:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:13:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 06:13:30:[cuda:0] Epoch 1200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 06:13:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 06:13:37:mr: 4317.211 INFO:Trainer:2024-05-24 06:13:37:mrr: 0.117 INFO:Trainer:2024-05-24 06:13:37:hits@1: 0.093 INFO:Trainer:2024-05-24 06:13:37:hits@3: 0.125 INFO:Trainer:2024-05-24 06:13:37:hits@10: 0.163 INFO:Trainer:2024-05-24 06:13:37:mr_type: 350.183 INFO:Trainer:2024-05-24 06:13:37:mrr_type: 0.207 INFO:Trainer:2024-05-24 06:13:37:hits@1_type: 0.153 INFO:Trainer:2024-05-24 06:13:37:hits@3_type: 0.22 INFO:Trainer:2024-05-24 06:13:37:hits@10_type: 0.311 INFO:EarlyStopping:2024-05-24 06:13:37:Validation score improved (0.158000 --> 0.163000). Saving model ... INFO:Trainer:2024-05-24 06:13:37:[cuda:0] Epoch [1200/10000] | Batchsize: 1024 | loss: 0.350309 | 7.48303 seconds/epoch DEBUG:connectionpool:2024-05-24 06:13:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:13:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:13:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:13:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:13:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:14:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:14:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:14:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:14:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:14:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:14:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:14:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:14:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:14:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:14:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:14:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:15:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:15:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:15:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:15:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:15:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:15:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:15:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:15:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:15:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:15:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:15:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:15:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:16:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:16:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:16:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:16:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:16:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:16:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:16:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:16:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:16:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:16:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:16:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:17:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:17:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:17:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:17:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:17:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:17:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:17:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:17:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:17:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:17:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:17:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:18:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:18:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:18:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:18:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:18:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:18:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:18:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:18:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:18:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:18:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:18:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:18:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:19:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:19:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:19:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:19:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:19:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:19:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:19:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:19:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:19:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:19:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:19:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:20:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:20:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:20:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:20:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:20:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:20:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:20:31:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-24 06:20:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:20:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:20:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:20:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:20:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:20:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:21:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:21:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:21:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:21:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:21:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:21:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:21:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:21:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:21:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:21:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:21:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:22:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:22:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:22:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:22:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:22:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:22:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:22:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:22:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:22:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:22:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:22:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:23:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:23:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:23:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:23:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:23:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:23:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:23:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:23:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:23:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:23:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:23:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:23:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:24:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:24:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:24:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:24:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:24:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:24:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:24:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:24:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:24:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:24:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:24:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:25:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:25:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:25:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:25:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:25:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:25:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:25:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:25:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:25:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:25:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:25:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 06:25:56:[cuda:0] Epoch 1300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 06:25:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 06:26:02:mr: 4238.185 INFO:Trainer:2024-05-24 06:26:02:mrr: 0.119 INFO:Trainer:2024-05-24 06:26:02:hits@1: 0.095 INFO:Trainer:2024-05-24 06:26:02:hits@3: 0.128 INFO:Trainer:2024-05-24 06:26:02:hits@10: 0.167 INFO:Trainer:2024-05-24 06:26:02:mr_type: 348.749 INFO:Trainer:2024-05-24 06:26:02:mrr_type: 0.209 INFO:Trainer:2024-05-24 06:26:02:hits@1_type: 0.155 INFO:Trainer:2024-05-24 06:26:02:hits@3_type: 0.222 INFO:Trainer:2024-05-24 06:26:02:hits@10_type: 0.314 INFO:EarlyStopping:2024-05-24 06:26:02:Validation score improved (0.163000 --> 0.167000). Saving model ... INFO:Trainer:2024-05-24 06:26:02:[cuda:0] Epoch [1300/10000] | Batchsize: 1024 | loss: 0.348474 | 7.48109 seconds/epoch DEBUG:connectionpool:2024-05-24 06:26:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:26:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:26:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:26:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:26:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:26:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:26:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:26:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:26:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:26:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:26:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:27:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:27:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:27:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:27:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:27:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:27:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:27:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:27:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:27:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:27:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:27:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:28:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:28:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:28:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:28:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:28:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:28:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:28:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:28:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:28:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:28:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:28:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:28:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:29:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:29:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:29:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:29:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:29:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:29:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:29:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:29:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:29:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:29:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:29:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:30:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:30:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:30:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:30:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:30:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:30:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:30:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:30:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:30:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:30:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:30:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:30:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:31:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:31:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:31:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:31:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:31:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:31:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:31:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:31:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:31:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:31:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:31:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:32:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:32:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:32:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:32:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:32:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:32:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:32:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:32:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:32:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:32:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:32:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:32:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:33:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:33:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:33:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:33:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:33:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:33:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:33:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:33:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:33:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:33:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:33:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:34:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:34:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:34:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:34:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:34:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:34:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:34:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:34:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:34:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:34:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:34:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:35:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:35:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:35:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:35:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:35:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:35:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:35:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:35:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:35:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:35:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:35:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:35:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:36:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:36:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:36:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:36:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:36:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:36:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:36:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:36:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:36:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:36:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:36:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:37:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:37:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:37:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:37:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:37:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:37:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:37:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:37:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:37:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:37:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:37:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:37:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:38:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:38:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:38:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 06:38:18:[cuda:0] Epoch 1400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 06:38:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 06:38:24:mr: 4164.56 INFO:Trainer:2024-05-24 06:38:24:mrr: 0.122 INFO:Trainer:2024-05-24 06:38:24:hits@1: 0.097 INFO:Trainer:2024-05-24 06:38:24:hits@3: 0.13 INFO:Trainer:2024-05-24 06:38:24:hits@10: 0.17 INFO:Trainer:2024-05-24 06:38:24:mr_type: 347.327 INFO:Trainer:2024-05-24 06:38:24:mrr_type: 0.211 INFO:Trainer:2024-05-24 06:38:24:hits@1_type: 0.156 INFO:Trainer:2024-05-24 06:38:24:hits@3_type: 0.224 INFO:Trainer:2024-05-24 06:38:24:hits@10_type: 0.315 INFO:EarlyStopping:2024-05-24 06:38:24:Validation score improved (0.167000 --> 0.170000). Saving model ... DEBUG:connectionpool:2024-05-24 06:38:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 06:38:24:[cuda:0] Epoch [1400/10000] | Batchsize: 1024 | loss: 0.347351 | 7.47654 seconds/epoch DEBUG:connectionpool:2024-05-24 06:38:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:38:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:38:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:38:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:38:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:38:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:39:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:39:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:39:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:39:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:39:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:39:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:39:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:39:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:39:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:39:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:39:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:39:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:40:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:40:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:40:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:40:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:40:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:40:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:40:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:40:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:40:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:40:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:40:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:41:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:41:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:41:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:41:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:41:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:41:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:41:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:41:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:41:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:41:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:41:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:41:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:42:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:42:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:42:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:42:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:42:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:42:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:42:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:42:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:42:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:42:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:42:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:43:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:43:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:43:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:43:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:43:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:43:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:43:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:43:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:43:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:43:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:43:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:43:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:44:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:44:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:44:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:44:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:44:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:44:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:44:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:44:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:44:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:44:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:44:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:45:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:45:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:45:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:45:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:45:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:45:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:45:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:45:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:45:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:45:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:45:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:45:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:46:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:46:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:46:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:46:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:46:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:46:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:46:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:46:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:46:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:46:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:46:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:47:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:47:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:47:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:47:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:47:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:47:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:47:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:47:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:47:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:47:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:47:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:47:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:48:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:48:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:48:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:48:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:48:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:48:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:48:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:48:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:48:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:48:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:48:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:49:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:49:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:49:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:49:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:49:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:49:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:49:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:49:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:49:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:49:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:49:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:49:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:50:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:50:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:50:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:50:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:50:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:50:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:50:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 06:50:40:[cuda:0] Epoch 1500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 06:50:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 06:50:46:mr: 4094.824 INFO:Trainer:2024-05-24 06:50:46:mrr: 0.124 INFO:Trainer:2024-05-24 06:50:46:hits@1: 0.098 INFO:Trainer:2024-05-24 06:50:46:hits@3: 0.132 INFO:Trainer:2024-05-24 06:50:46:hits@10: 0.173 INFO:Trainer:2024-05-24 06:50:46:mr_type: 346.19 INFO:Trainer:2024-05-24 06:50:46:mrr_type: 0.213 INFO:Trainer:2024-05-24 06:50:46:hits@1_type: 0.158 INFO:Trainer:2024-05-24 06:50:46:hits@3_type: 0.225 INFO:Trainer:2024-05-24 06:50:46:hits@10_type: 0.317 INFO:EarlyStopping:2024-05-24 06:50:46:Validation score improved (0.170000 --> 0.173000). Saving model ... DEBUG:connectionpool:2024-05-24 06:50:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 06:50:46:[cuda:0] Epoch [1500/10000] | Batchsize: 1024 | loss: 0.345965 | 7.47313 seconds/epoch DEBUG:connectionpool:2024-05-24 06:50:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:50:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:51:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:51:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:51:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:51:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:51:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:51:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:51:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:51:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:51:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:51:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:51:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:51:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:52:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:52:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:52:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:52:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:52:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:52:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:52:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:52:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:52:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:52:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:52:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:53:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:53:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:53:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:53:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:53:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:53:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:53:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:53:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:53:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:53:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:53:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:54:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:54:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:54:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:54:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:54:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:54:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:54:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:54:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:54:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:54:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:54:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:54:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:55:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:55:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:55:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:55:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:55:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:55:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:55:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:55:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:55:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:55:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:55:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:56:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:56:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:56:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:56:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:56:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:56:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:56:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:56:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:56:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:56:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:56:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:56:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:57:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:57:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:57:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:57:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:57:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:57:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:57:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:57:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:57:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:57:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:57:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:58:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:58:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:58:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:58:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:58:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:58:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:58:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:58:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:58:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:58:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:58:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:58:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:59:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:59:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:59:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:59:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:59:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:59:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:59:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:59:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:59:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:59:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 06:59:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:00:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:00:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:00:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:00:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:00:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:00:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:00:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:00:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:00:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:00:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:00:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:00:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:01:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:01:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:01:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:01:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:01:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:01:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:01:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:01:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:01:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:01:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:01:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:02:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:02:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:02:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:02:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:02:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:02:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:02:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:02:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:02:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:02:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:02:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:02:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 07:03:02:[cuda:0] Epoch 1600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 07:03:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:03:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 07:03:09:mr: 4029.046 INFO:Trainer:2024-05-24 07:03:09:mrr: 0.126 INFO:Trainer:2024-05-24 07:03:09:hits@1: 0.1 INFO:Trainer:2024-05-24 07:03:09:hits@3: 0.134 INFO:Trainer:2024-05-24 07:03:09:hits@10: 0.176 INFO:Trainer:2024-05-24 07:03:09:mr_type: 345.092 INFO:Trainer:2024-05-24 07:03:09:mrr_type: 0.215 INFO:Trainer:2024-05-24 07:03:09:hits@1_type: 0.159 INFO:Trainer:2024-05-24 07:03:09:hits@3_type: 0.226 INFO:Trainer:2024-05-24 07:03:09:hits@10_type: 0.319 INFO:EarlyStopping:2024-05-24 07:03:09:Validation score improved (0.173000 --> 0.176000). Saving model ... INFO:Trainer:2024-05-24 07:03:10:[cuda:0] Epoch [1600/10000] | Batchsize: 1024 | loss: 0.344930 | 7.46985 seconds/epoch DEBUG:connectionpool:2024-05-24 07:03:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:03:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:03:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:03:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:03:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:03:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:03:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:03:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:03:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:04:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:04:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:04:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:04:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:04:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:04:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:04:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:04:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:04:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:04:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:04:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:04:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:05:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:05:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:05:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:05:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:05:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:05:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:05:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:05:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:05:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:05:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:05:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:06:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:06:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:06:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:06:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:06:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:06:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:06:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:06:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:06:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:06:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:06:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:06:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:07:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:07:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:07:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:07:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:07:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:07:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:07:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:07:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:07:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:07:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:07:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:08:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:08:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:08:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:08:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:08:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:08:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:08:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:08:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:08:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:08:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:08:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:08:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:09:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:09:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:09:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:09:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:09:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:09:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:09:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:09:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:09:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:09:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:09:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:10:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:10:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:10:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:10:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:10:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:10:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:10:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:10:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:10:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:10:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:10:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:10:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:11:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:11:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:11:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:11:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:11:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:11:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:11:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:11:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:11:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:11:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:11:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:12:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:12:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:12:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:12:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:12:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:12:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:12:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:12:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:12:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:12:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:12:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:13:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:13:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:13:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:13:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:13:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:13:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:13:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:13:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:13:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:13:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:13:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:13:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:14:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:14:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:14:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:14:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:14:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:14:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:14:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:14:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:14:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:14:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:14:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:15:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:15:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:15:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:15:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:15:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:15:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:15:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 07:15:32:[cuda:0] Epoch 1700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 07:15:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 07:15:38:mr: 3965.434 INFO:Trainer:2024-05-24 07:15:38:mrr: 0.128 INFO:Trainer:2024-05-24 07:15:38:hits@1: 0.101 INFO:Trainer:2024-05-24 07:15:38:hits@3: 0.137 INFO:Trainer:2024-05-24 07:15:38:hits@10: 0.18 INFO:Trainer:2024-05-24 07:15:38:mr_type: 343.962 INFO:Trainer:2024-05-24 07:15:38:mrr_type: 0.215 INFO:Trainer:2024-05-24 07:15:38:hits@1_type: 0.159 INFO:Trainer:2024-05-24 07:15:38:hits@3_type: 0.228 INFO:Trainer:2024-05-24 07:15:38:hits@10_type: 0.322 INFO:EarlyStopping:2024-05-24 07:15:38:Validation score improved (0.176000 --> 0.180000). Saving model ... INFO:Trainer:2024-05-24 07:15:38:[cuda:0] Epoch [1700/10000] | Batchsize: 1024 | loss: 0.343715 | 7.47140 seconds/epoch DEBUG:connectionpool:2024-05-24 07:15:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:15:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:15:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:15:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:16:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:16:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:16:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:16:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:16:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:16:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:16:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:16:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:16:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:16:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:16:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:17:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:17:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:17:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:17:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:17:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:17:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:17:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:17:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:17:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:17:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:17:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:17:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:18:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:18:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:18:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:18:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:18:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:18:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:18:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:18:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:18:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:18:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:18:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:19:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:19:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:19:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:19:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:19:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:19:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:19:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:19:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:19:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:19:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:19:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:19:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:20:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:20:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:20:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:20:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:20:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:20:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:20:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:20:40:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-24 07:20:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:20:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:20:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:20:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:21:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:21:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:21:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:21:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:21:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:21:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:21:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:21:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:21:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:21:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:21:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:22:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:22:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:22:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:22:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:22:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:22:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:22:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:22:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:22:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:22:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:22:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:22:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:23:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:23:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:23:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:23:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:23:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:23:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:23:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:23:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:23:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:23:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:23:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:24:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:24:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:24:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:24:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:24:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:24:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:24:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:24:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:24:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:24:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:24:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:24:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:25:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:25:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:25:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:25:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:25:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:25:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:25:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:25:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:25:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:25:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:25:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:26:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:26:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:26:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:26:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:26:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:26:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:26:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:26:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:26:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:26:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:26:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:26:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:27:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:27:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:27:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:27:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:27:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:27:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:27:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:27:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:27:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:27:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 07:27:55:[cuda:0] Epoch 1800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 07:27:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 07:28:01:mr: 3908.66 INFO:Trainer:2024-05-24 07:28:01:mrr: 0.131 INFO:Trainer:2024-05-24 07:28:01:hits@1: 0.103 INFO:Trainer:2024-05-24 07:28:01:hits@3: 0.139 INFO:Trainer:2024-05-24 07:28:01:hits@10: 0.183 INFO:Trainer:2024-05-24 07:28:01:mr_type: 343.052 INFO:Trainer:2024-05-24 07:28:01:mrr_type: 0.217 INFO:Trainer:2024-05-24 07:28:01:hits@1_type: 0.16 INFO:Trainer:2024-05-24 07:28:01:hits@3_type: 0.23 INFO:Trainer:2024-05-24 07:28:01:hits@10_type: 0.325 INFO:EarlyStopping:2024-05-24 07:28:01:Validation score improved (0.180000 --> 0.183000). Saving model ... INFO:Trainer:2024-05-24 07:28:01:[cuda:0] Epoch [1800/10000] | Batchsize: 1024 | loss: 0.342257 | 7.46907 seconds/epoch DEBUG:connectionpool:2024-05-24 07:28:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:28:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:28:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:28:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:28:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:28:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:28:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:28:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:28:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:28:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:28:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:29:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:29:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:29:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:29:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:29:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:29:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:29:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:29:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:29:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:29:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:29:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:29:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:30:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:30:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:30:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:30:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:30:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:30:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:30:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:30:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:30:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:30:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:30:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:31:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:31:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:31:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:31:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:31:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:31:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:31:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:31:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:31:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:31:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:31:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:31:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:32:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:32:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:32:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:32:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:32:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:32:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:32:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:32:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:32:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:32:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:32:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:33:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:33:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:33:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:33:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:33:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:33:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:33:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:33:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:33:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:33:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:33:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:34:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:34:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:34:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:34:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:34:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:34:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:34:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:34:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:34:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:34:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:34:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:34:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:35:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:35:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:35:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:35:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:35:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:35:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:35:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:35:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:35:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:35:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:35:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:36:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:36:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:36:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:36:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:36:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:36:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:36:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:36:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:36:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:36:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:36:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:36:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:37:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:37:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:37:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:37:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:37:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:37:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:37:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:37:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:37:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:37:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:37:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:38:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:38:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:38:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:38:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:38:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:38:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:38:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:38:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:38:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:38:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:38:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:39:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:39:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:39:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:39:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:39:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:39:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:39:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:39:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:39:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:39:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:39:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:39:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:40:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:40:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:40:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 07:40:17:[cuda:0] Epoch 1900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 07:40:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 07:40:23:mr: 3853.667 INFO:Trainer:2024-05-24 07:40:23:mrr: 0.133 INFO:Trainer:2024-05-24 07:40:23:hits@1: 0.105 INFO:Trainer:2024-05-24 07:40:23:hits@3: 0.142 INFO:Trainer:2024-05-24 07:40:23:hits@10: 0.187 INFO:Trainer:2024-05-24 07:40:23:mr_type: 342.078 INFO:Trainer:2024-05-24 07:40:23:mrr_type: 0.218 INFO:Trainer:2024-05-24 07:40:23:hits@1_type: 0.161 INFO:Trainer:2024-05-24 07:40:23:hits@3_type: 0.231 INFO:Trainer:2024-05-24 07:40:23:hits@10_type: 0.327 INFO:EarlyStopping:2024-05-24 07:40:23:Validation score improved (0.183000 --> 0.187000). Saving model ... INFO:Trainer:2024-05-24 07:40:24:[cuda:0] Epoch [1900/10000] | Batchsize: 1024 | loss: 0.341067 | 7.46666 seconds/epoch DEBUG:connectionpool:2024-05-24 07:40:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:40:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:40:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:40:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:40:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:40:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:40:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:41:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:41:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:41:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:41:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:41:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:41:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:41:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:41:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:41:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:41:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:41:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:41:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:42:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:42:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:42:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:42:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:42:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:42:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:42:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:42:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:42:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:42:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:42:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:43:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:43:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:43:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:43:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:43:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:43:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:43:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:43:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:43:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:43:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:43:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:44:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:44:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:44:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:44:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:44:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:44:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:44:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:44:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:44:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:44:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:44:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:44:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:45:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:45:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:45:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:45:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:45:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:45:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:45:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:45:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:45:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:45:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:45:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:46:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:46:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:46:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:46:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:46:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:46:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:46:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:46:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:46:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:46:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:46:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:47:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:47:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:47:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:47:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:47:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:47:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:47:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:47:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:47:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:47:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:47:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:47:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:48:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:48:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:48:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:48:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:48:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:48:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:48:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:48:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:48:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:49:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 502 332 DEBUG:connectionpool:2024-05-24 07:49:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:49:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:49:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:49:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:49:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:49:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:49:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:49:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:49:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:50:56:Starting new HTTPS connection (12): api.wandb.ai:443 DEBUG:connectionpool:2024-05-24 07:50:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:51:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:51:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:51:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:51:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:51:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:51:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:51:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:51:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:51:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:51:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:51:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:51:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:52:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:52:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:52:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:52:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:52:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:52:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:52:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 07:52:40:[cuda:0] Epoch 2000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 07:52:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 07:52:46:mr: 3804.539 INFO:Trainer:2024-05-24 07:52:46:mrr: 0.135 INFO:Trainer:2024-05-24 07:52:46:hits@1: 0.106 INFO:Trainer:2024-05-24 07:52:46:hits@3: 0.144 INFO:Trainer:2024-05-24 07:52:46:hits@10: 0.191 INFO:Trainer:2024-05-24 07:52:46:mr_type: 341.914 INFO:Trainer:2024-05-24 07:52:46:mrr_type: 0.219 INFO:Trainer:2024-05-24 07:52:46:hits@1_type: 0.162 INFO:Trainer:2024-05-24 07:52:46:hits@3_type: 0.231 INFO:Trainer:2024-05-24 07:52:46:hits@10_type: 0.328 INFO:EarlyStopping:2024-05-24 07:52:46:Validation score improved (0.187000 --> 0.191000). Saving model ... INFO:Trainer:2024-05-24 07:52:46:[cuda:0] Epoch [2000/10000] | Batchsize: 1024 | loss: 0.340111 | 7.46469 seconds/epoch DEBUG:connectionpool:2024-05-24 07:52:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:52:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:52:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:53:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:53:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:53:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:53:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:53:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:53:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:53:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:53:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:53:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:53:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:53:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:54:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:54:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:54:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:54:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:54:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:54:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:54:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:54:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:54:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:54:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:54:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:54:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:55:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:55:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:55:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:55:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:55:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:55:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:55:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:55:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:55:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:55:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:55:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:56:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:56:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:56:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:56:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:56:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:56:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:56:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:56:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:56:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:56:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:56:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:56:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:57:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:57:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:57:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:57:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:57:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:57:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:57:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:57:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:57:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:57:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:57:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:58:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:58:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:58:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:58:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:58:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:58:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:58:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:58:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:58:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:58:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:58:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:58:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:59:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:59:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:59:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:59:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:59:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:59:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:59:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:59:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:59:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:59:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 07:59:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:00:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:00:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:00:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:00:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:00:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:00:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:00:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:00:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:00:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:00:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:00:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:00:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:01:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:01:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:01:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:01:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:01:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:01:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:01:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:01:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:01:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:01:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:01:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:02:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:02:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:02:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:02:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:02:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:02:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:02:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:02:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:02:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:02:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:02:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:03:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:03:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:03:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:03:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:03:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:03:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:03:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:03:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:03:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:03:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:03:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:03:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:04:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:04:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:04:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:04:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:04:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:04:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:04:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:04:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:04:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:04:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:04:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:05:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 08:05:02:[cuda:0] Epoch 2100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 08:05:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 08:05:08:mr: 3752.133 INFO:Trainer:2024-05-24 08:05:08:mrr: 0.137 INFO:Trainer:2024-05-24 08:05:08:hits@1: 0.108 INFO:Trainer:2024-05-24 08:05:08:hits@3: 0.146 INFO:Trainer:2024-05-24 08:05:08:hits@10: 0.194 INFO:Trainer:2024-05-24 08:05:08:mr_type: 340.948 INFO:Trainer:2024-05-24 08:05:08:mrr_type: 0.22 INFO:Trainer:2024-05-24 08:05:08:hits@1_type: 0.163 INFO:Trainer:2024-05-24 08:05:08:hits@3_type: 0.233 INFO:Trainer:2024-05-24 08:05:08:hits@10_type: 0.331 INFO:EarlyStopping:2024-05-24 08:05:08:Validation score improved (0.191000 --> 0.194000). Saving model ... INFO:Trainer:2024-05-24 08:05:08:[cuda:0] Epoch [2100/10000] | Batchsize: 1024 | loss: 0.338917 | 7.46250 seconds/epoch DEBUG:connectionpool:2024-05-24 08:05:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:05:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:05:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:05:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:05:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:05:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:05:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:05:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:05:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:05:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:06:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:06:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:06:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:06:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:06:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:06:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:06:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:06:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:06:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:06:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:06:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:07:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:07:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:07:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:07:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:07:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:07:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:07:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:07:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:07:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:07:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:07:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:07:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:08:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:08:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:08:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:08:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:08:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:08:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:08:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:08:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:08:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:08:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:08:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:09:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:09:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:09:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:09:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:09:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:09:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:09:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:09:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:09:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:09:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:09:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:09:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:10:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:10:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:10:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:10:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:10:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:10:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:10:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:10:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:10:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:10:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:10:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:11:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:11:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:11:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:11:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:11:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:11:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:11:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:11:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:11:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:11:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:11:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:12:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:12:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:12:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:12:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:12:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:12:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:12:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:12:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:12:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:12:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:12:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:12:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:13:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:13:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:13:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:13:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:13:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:13:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:13:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:13:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:13:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:13:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:13:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:14:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:14:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:14:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:14:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:14:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:14:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:14:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:14:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:14:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:14:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:14:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:14:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:15:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:15:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:15:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:15:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:15:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:15:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:15:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:15:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:15:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:15:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:15:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:16:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:16:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:16:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:16:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:16:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:16:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:16:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:16:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:16:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:16:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:16:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:16:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:17:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:17:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:17:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:17:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:17:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 08:17:28:[cuda:0] Epoch 2200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 08:17:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 08:17:34:mr: 3694.688 INFO:Trainer:2024-05-24 08:17:34:mrr: 0.138 INFO:Trainer:2024-05-24 08:17:34:hits@1: 0.109 INFO:Trainer:2024-05-24 08:17:34:hits@3: 0.148 INFO:Trainer:2024-05-24 08:17:34:hits@10: 0.197 INFO:Trainer:2024-05-24 08:17:34:mr_type: 339.941 INFO:Trainer:2024-05-24 08:17:34:mrr_type: 0.22 INFO:Trainer:2024-05-24 08:17:34:hits@1_type: 0.163 INFO:Trainer:2024-05-24 08:17:34:hits@3_type: 0.233 INFO:Trainer:2024-05-24 08:17:34:hits@10_type: 0.332 INFO:EarlyStopping:2024-05-24 08:17:34:Validation score improved (0.194000 --> 0.197000). Saving model ... INFO:Trainer:2024-05-24 08:17:34:[cuda:0] Epoch [2200/10000] | Batchsize: 1024 | loss: 0.338102 | 7.46260 seconds/epoch DEBUG:connectionpool:2024-05-24 08:17:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:17:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:17:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:17:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:17:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:18:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:18:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:18:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:18:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:18:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:18:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:18:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:18:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:18:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:18:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:18:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:18:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:19:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:19:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:19:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:19:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:19:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:19:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:19:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:19:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:19:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:19:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:19:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:20:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:20:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:20:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:20:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:20:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:20:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:20:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:20:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:20:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:20:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:20:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:20:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:21:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:21:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:21:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:21:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:21:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:21:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:21:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:21:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:21:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:21:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:21:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:22:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:22:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:22:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:22:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:22:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:22:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:22:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:22:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:22:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:22:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:22:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:22:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:23:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:23:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:23:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:23:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:23:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:23:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:23:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:23:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:23:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:23:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:23:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:24:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:24:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:24:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:24:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:24:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:24:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:24:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:24:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:24:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:24:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:24:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:25:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:25:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:25:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:25:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:25:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:25:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:25:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:25:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:25:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:25:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:25:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:25:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:26:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:26:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:26:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:26:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:26:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:26:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:26:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:26:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:26:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:26:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:26:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:27:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:27:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:27:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:27:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:27:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:27:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:27:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:27:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:27:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:27:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:27:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:27:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:28:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:28:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:28:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:28:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:28:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:28:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:28:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:28:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:28:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:28:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:28:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:29:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:29:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:29:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:29:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:29:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:29:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:29:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:29:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:29:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:29:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 08:29:51:[cuda:0] Epoch 2300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 08:29:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 08:29:57:mr: 3644.217 INFO:Trainer:2024-05-24 08:29:57:mrr: 0.14 INFO:Trainer:2024-05-24 08:29:57:hits@1: 0.11 INFO:Trainer:2024-05-24 08:29:57:hits@3: 0.149 INFO:Trainer:2024-05-24 08:29:57:hits@10: 0.199 INFO:Trainer:2024-05-24 08:29:57:mr_type: 338.829 INFO:Trainer:2024-05-24 08:29:57:mrr_type: 0.221 INFO:Trainer:2024-05-24 08:29:57:hits@1_type: 0.164 INFO:Trainer:2024-05-24 08:29:57:hits@3_type: 0.235 INFO:Trainer:2024-05-24 08:29:57:hits@10_type: 0.334 INFO:EarlyStopping:2024-05-24 08:29:57:Validation score improved (0.197000 --> 0.199000). Saving model ... INFO:Trainer:2024-05-24 08:29:57:[cuda:0] Epoch [2300/10000] | Batchsize: 1024 | loss: 0.336938 | 7.46087 seconds/epoch DEBUG:connectionpool:2024-05-24 08:29:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:30:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:30:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:30:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:30:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:30:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:30:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:30:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:30:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:30:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:30:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:30:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:31:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:31:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:31:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:31:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:31:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:31:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:31:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:31:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:31:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:31:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:31:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:31:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:32:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:32:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:32:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:32:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:32:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:32:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:32:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:32:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:32:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:32:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:32:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:33:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:33:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:33:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:33:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:33:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:33:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:33:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:33:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:33:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:33:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:33:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:33:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:34:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:34:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:34:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:34:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:34:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:34:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:34:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:34:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:34:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:34:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:34:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:35:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:35:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:35:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:35:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:35:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:35:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:35:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:35:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:35:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:35:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:35:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:36:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:36:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:36:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:36:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:36:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:36:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:36:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:36:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:36:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:36:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:36:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:36:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:37:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:37:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:37:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:37:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:37:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:37:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:37:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:37:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:37:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:37:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:37:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:38:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:38:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:38:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:38:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:38:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:38:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:38:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:38:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:38:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:38:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:38:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:38:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:39:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:39:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:39:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:39:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:39:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:39:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:39:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:39:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:39:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:39:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:39:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:40:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:40:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:40:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:40:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:40:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:40:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:40:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:40:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:40:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:40:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:40:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:40:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:41:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:41:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:41:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:41:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:41:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:41:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:41:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:41:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:41:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:41:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:41:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:42:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:42:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:42:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:42:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:42:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:42:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 08:42:29:[cuda:0] Epoch 2400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 08:42:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 08:42:35:mr: 3594.668 INFO:Trainer:2024-05-24 08:42:35:mrr: 0.141 INFO:Trainer:2024-05-24 08:42:35:hits@1: 0.11 INFO:Trainer:2024-05-24 08:42:35:hits@3: 0.151 INFO:Trainer:2024-05-24 08:42:35:hits@10: 0.201 INFO:Trainer:2024-05-24 08:42:35:mr_type: 337.99 INFO:Trainer:2024-05-24 08:42:35:mrr_type: 0.222 INFO:Trainer:2024-05-24 08:42:35:hits@1_type: 0.165 INFO:Trainer:2024-05-24 08:42:35:hits@3_type: 0.236 INFO:Trainer:2024-05-24 08:42:35:hits@10_type: 0.335 INFO:EarlyStopping:2024-05-24 08:42:35:Validation score improved (0.199000 --> 0.201000). Saving model ... INFO:Trainer:2024-05-24 08:42:35:[cuda:0] Epoch [2400/10000] | Batchsize: 1024 | loss: 0.336114 | 7.46582 seconds/epoch DEBUG:connectionpool:2024-05-24 08:42:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:42:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:42:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:42:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:42:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:43:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:43:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:43:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:43:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:43:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:43:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:43:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:43:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:43:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:43:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:43:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:44:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:44:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:44:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:44:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:44:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:44:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:44:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:44:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:44:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:44:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:44:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:44:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:45:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:45:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:45:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:45:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:45:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:45:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:45:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:45:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:45:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:45:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:45:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:46:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:46:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:46:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:46:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:46:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:46:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:46:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:46:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:46:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:46:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:46:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:47:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:47:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:47:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:47:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:47:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:47:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:47:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:47:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:47:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:47:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:47:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:47:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:48:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:48:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:48:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:48:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:48:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:48:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:48:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:48:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:48:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:48:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:48:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:49:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:49:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:49:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:49:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:49:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:49:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:49:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:49:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:49:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:49:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:49:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:49:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:50:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:50:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:50:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:50:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:50:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:50:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:50:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:50:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:50:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:50:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:50:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:51:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:51:07:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-24 08:51:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:51:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:51:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:51:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:51:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:51:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:51:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:51:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:51:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:51:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:52:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:52:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:52:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:52:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:52:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:52:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:52:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:52:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:52:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:52:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:52:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:52:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:53:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:53:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:53:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:53:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:53:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:53:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:53:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:53:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:53:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:53:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:53:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:54:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:54:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:54:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:54:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:54:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:54:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:54:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:54:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:54:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:54:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 08:54:51:[cuda:0] Epoch 2500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 08:54:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 08:54:57:mr: 3545.176 INFO:Trainer:2024-05-24 08:54:57:mrr: 0.142 INFO:Trainer:2024-05-24 08:54:57:hits@1: 0.111 INFO:Trainer:2024-05-24 08:54:57:hits@3: 0.153 INFO:Trainer:2024-05-24 08:54:57:hits@10: 0.203 INFO:Trainer:2024-05-24 08:54:57:mr_type: 337.119 INFO:Trainer:2024-05-24 08:54:57:mrr_type: 0.223 INFO:Trainer:2024-05-24 08:54:57:hits@1_type: 0.166 INFO:Trainer:2024-05-24 08:54:57:hits@3_type: 0.237 INFO:Trainer:2024-05-24 08:54:57:hits@10_type: 0.335 INFO:EarlyStopping:2024-05-24 08:54:57:Validation score improved (0.201000 --> 0.203000). Saving model ... INFO:Trainer:2024-05-24 08:54:58:[cuda:0] Epoch [2500/10000] | Batchsize: 1024 | loss: 0.335063 | 7.46421 seconds/epoch DEBUG:connectionpool:2024-05-24 08:55:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:55:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:55:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:55:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:55:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:55:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:55:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:55:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:55:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:55:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:55:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:55:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:56:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:56:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:56:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:56:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:56:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:56:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:56:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:56:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:56:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:56:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:56:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:57:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:57:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:57:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:57:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:57:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:57:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:57:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:57:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:57:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:57:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:57:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:58:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:58:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:58:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:58:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:58:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:58:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:58:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:58:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:58:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:58:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:58:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:58:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:59:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:59:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:59:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:59:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:59:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:59:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:59:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:59:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:59:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:59:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 08:59:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:00:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:00:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:00:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:00:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:00:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:00:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:00:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:00:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:00:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:00:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:00:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:00:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:01:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:01:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:01:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:01:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:01:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:01:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:01:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:01:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:01:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:01:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:01:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:02:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:02:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:02:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:02:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:02:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:02:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:02:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:02:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:02:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:02:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:02:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:03:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:03:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:03:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:03:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:03:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:03:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:03:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:03:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:03:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:03:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:03:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:03:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:04:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:04:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:04:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:04:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:04:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:04:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:04:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:04:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:04:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:04:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:04:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:05:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:05:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:05:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:05:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:05:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:05:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:05:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:05:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:05:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:05:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:05:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:06:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:06:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:06:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:06:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:06:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:06:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:06:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:06:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:06:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:06:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:06:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:06:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:07:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:07:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:07:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 09:07:16:[cuda:0] Epoch 2600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 09:07:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 09:07:22:mr: 3499.366 INFO:Trainer:2024-05-24 09:07:22:mrr: 0.143 INFO:Trainer:2024-05-24 09:07:22:hits@1: 0.111 INFO:Trainer:2024-05-24 09:07:22:hits@3: 0.154 INFO:Trainer:2024-05-24 09:07:22:hits@10: 0.205 INFO:Trainer:2024-05-24 09:07:22:mr_type: 336.111 INFO:Trainer:2024-05-24 09:07:22:mrr_type: 0.224 INFO:Trainer:2024-05-24 09:07:22:hits@1_type: 0.166 INFO:Trainer:2024-05-24 09:07:22:hits@3_type: 0.238 INFO:Trainer:2024-05-24 09:07:22:hits@10_type: 0.336 INFO:EarlyStopping:2024-05-24 09:07:22:Validation score improved (0.203000 --> 0.205000). Saving model ... INFO:Trainer:2024-05-24 09:07:22:[cuda:0] Epoch [2600/10000] | Batchsize: 1024 | loss: 0.334528 | 7.46346 seconds/epoch DEBUG:connectionpool:2024-05-24 09:07:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:07:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:07:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:07:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:07:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:07:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:07:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:08:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:08:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:08:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:08:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:08:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:08:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:08:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:08:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:08:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:08:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:08:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:09:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:09:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:09:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:09:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:09:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:09:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:09:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:09:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:09:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:09:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:09:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:09:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:10:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:10:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:10:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:10:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:10:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:10:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:10:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:10:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:10:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:10:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:10:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:11:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:11:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:11:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:11:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:11:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:11:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:11:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:11:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:11:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:11:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:11:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:12:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:12:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:12:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:12:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:12:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:12:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:12:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:12:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:12:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:12:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:12:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:12:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:13:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:13:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:13:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:13:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:13:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:13:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:13:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:13:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:13:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:13:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:13:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:14:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:14:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:14:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:14:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:14:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:14:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:14:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:14:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:14:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:14:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:14:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:14:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:15:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:15:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:15:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:15:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:15:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:15:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:15:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:15:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:15:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:15:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:15:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:16:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:16:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:16:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:16:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:16:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:16:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:16:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:16:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:16:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:16:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:16:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:17:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:17:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:17:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:17:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:17:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:17:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:17:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:17:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:17:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:17:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:17:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:17:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:18:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:18:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:18:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:18:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:18:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:18:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:18:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:18:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:18:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:18:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:18:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:19:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:19:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:19:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:19:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:19:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:19:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:19:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:19:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 09:19:40:[cuda:0] Epoch 2700 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 09:19:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 09:19:46:mr: 3452.249 INFO:Trainer:2024-05-24 09:19:46:mrr: 0.144 INFO:Trainer:2024-05-24 09:19:46:hits@1: 0.112 INFO:Trainer:2024-05-24 09:19:46:hits@3: 0.155 INFO:Trainer:2024-05-24 09:19:46:hits@10: 0.207 INFO:Trainer:2024-05-24 09:19:46:mr_type: 335.327 INFO:Trainer:2024-05-24 09:19:46:mrr_type: 0.224 INFO:Trainer:2024-05-24 09:19:46:hits@1_type: 0.166 INFO:Trainer:2024-05-24 09:19:46:hits@3_type: 0.239 INFO:Trainer:2024-05-24 09:19:46:hits@10_type: 0.337 INFO:EarlyStopping:2024-05-24 09:19:46:Validation score improved (0.205000 --> 0.207000). Saving model ... INFO:Trainer:2024-05-24 09:19:46:[cuda:0] Epoch [2700/10000] | Batchsize: 1024 | loss: 0.333787 | 7.46278 seconds/epoch DEBUG:connectionpool:2024-05-24 09:19:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:19:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:20:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:20:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:20:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:20:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:20:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:20:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:20:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:20:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:20:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:20:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:20:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:20:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:21:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:21:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:21:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:21:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:21:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:21:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:21:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:21:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:21:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:21:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:21:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:22:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:22:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:22:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:22:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:22:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:22:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:22:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:22:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:22:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:22:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:22:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:23:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:23:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:23:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:23:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:23:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:23:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:23:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:23:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:23:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:23:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:23:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:23:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:24:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:24:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:24:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:24:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:24:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:24:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:24:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:24:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:24:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:24:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:24:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:25:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:25:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:25:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:25:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:25:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:25:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:25:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:25:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:25:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:25:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:25:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:26:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:26:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:26:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:26:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:26:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:26:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:26:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:26:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:26:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:26:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:26:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:26:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:27:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:27:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:27:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:27:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:27:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:27:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:27:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:27:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:27:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:27:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:27:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:28:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:28:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:28:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:28:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:28:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:28:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:28:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:28:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:28:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:28:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:28:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:28:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:29:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:29:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:29:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:29:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:29:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:29:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:29:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:29:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:29:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:29:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:29:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:30:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:30:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:30:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:30:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:30:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:30:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:30:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:30:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:30:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:30:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:30:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:31:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:31:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:31:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:31:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:31:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:31:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:31:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:31:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:31:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:31:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:31:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:31:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 09:32:00:[cuda:0] Epoch 2800 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 09:32:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 09:32:06:mr: 3406.532 INFO:Trainer:2024-05-24 09:32:06:mrr: 0.145 INFO:Trainer:2024-05-24 09:32:06:hits@1: 0.113 INFO:Trainer:2024-05-24 09:32:06:hits@3: 0.156 INFO:Trainer:2024-05-24 09:32:06:hits@10: 0.208 INFO:Trainer:2024-05-24 09:32:06:mr_type: 334.482 INFO:Trainer:2024-05-24 09:32:06:mrr_type: 0.225 INFO:Trainer:2024-05-24 09:32:06:hits@1_type: 0.166 INFO:Trainer:2024-05-24 09:32:06:hits@3_type: 0.239 INFO:Trainer:2024-05-24 09:32:06:hits@10_type: 0.338 INFO:EarlyStopping:2024-05-24 09:32:06:Validation score improved (0.207000 --> 0.208000). Saving model ... INFO:Trainer:2024-05-24 09:32:06:[cuda:0] Epoch [2800/10000] | Batchsize: 1024 | loss: 0.332595 | 7.46046 seconds/epoch DEBUG:connectionpool:2024-05-24 09:32:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:32:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:32:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:32:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:32:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:32:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:32:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:32:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:32:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:32:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:33:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:33:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:33:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:33:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:33:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:33:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:33:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:33:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:33:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:33:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:33:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:34:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:34:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:34:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:34:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:34:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:34:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:34:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:34:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:34:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:34:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:34:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:34:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:35:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:35:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:35:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:35:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:35:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:35:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:35:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:35:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:35:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:35:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:35:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:36:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:36:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:36:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:36:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:36:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:36:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:36:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:36:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:36:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:36:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:36:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:37:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:37:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:37:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:37:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:37:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:37:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:37:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:37:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:37:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:37:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:37:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:37:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:38:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:38:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:38:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:38:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:38:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:38:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:38:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:38:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:38:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:38:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:38:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:39:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:39:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:39:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:39:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:39:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:39:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:39:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:39:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:39:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:39:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:39:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:40:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:40:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:40:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:40:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:40:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:40:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:40:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:40:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:40:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:40:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:40:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:40:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:41:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:41:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:41:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:41:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:41:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:41:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:41:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:41:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:41:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:41:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:41:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:42:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:42:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:42:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:42:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:42:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:42:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:42:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:42:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:42:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:42:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:42:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:43:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:43:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:43:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:43:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:43:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:43:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:43:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:43:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:43:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:43:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:43:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:43:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:44:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:44:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:44:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:44:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 09:44:21:[cuda:0] Epoch 2900 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 09:44:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 09:44:27:mr: 3359.481 INFO:Trainer:2024-05-24 09:44:27:mrr: 0.147 INFO:Trainer:2024-05-24 09:44:27:hits@1: 0.114 INFO:Trainer:2024-05-24 09:44:27:hits@3: 0.158 INFO:Trainer:2024-05-24 09:44:27:hits@10: 0.21 INFO:Trainer:2024-05-24 09:44:27:mr_type: 333.699 INFO:Trainer:2024-05-24 09:44:27:mrr_type: 0.225 INFO:Trainer:2024-05-24 09:44:27:hits@1_type: 0.166 INFO:Trainer:2024-05-24 09:44:27:hits@3_type: 0.239 INFO:Trainer:2024-05-24 09:44:27:hits@10_type: 0.338 INFO:EarlyStopping:2024-05-24 09:44:27:Validation score improved (0.208000 --> 0.210000). Saving model ... INFO:Trainer:2024-05-24 09:44:27:[cuda:0] Epoch [2900/10000] | Batchsize: 1024 | loss: 0.332011 | 7.45863 seconds/epoch DEBUG:connectionpool:2024-05-24 09:44:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:44:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:44:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:44:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:44:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:44:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:45:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:45:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:45:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:45:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:45:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:45:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:45:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:45:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:45:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:45:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:45:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:46:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:46:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:46:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:46:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:46:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:46:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:46:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:46:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:46:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:46:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:46:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:46:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:47:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:47:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:47:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:47:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:47:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:47:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:47:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:47:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:47:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:47:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:47:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:48:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:48:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:48:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:48:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:48:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:48:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:48:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:48:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:48:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:48:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:48:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:48:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:49:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:49:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:49:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:49:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:49:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:49:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:49:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:49:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:49:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:49:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:49:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:50:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:50:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:50:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:50:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:50:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:50:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:50:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:50:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:50:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:50:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:50:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:51:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:51:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:51:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:51:16:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-24 09:51:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:51:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:51:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:51:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:51:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:51:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:51:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:51:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:51:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:52:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:52:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:52:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:52:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:52:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:52:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:52:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:52:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:52:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:52:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:52:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:53:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:53:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:53:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:53:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:53:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:53:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:53:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:53:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:53:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:53:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:53:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:54:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:54:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:54:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:54:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:54:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:54:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:54:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:54:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:54:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:54:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:54:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:54:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:55:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:55:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:55:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:55:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:55:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:55:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:55:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:55:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:55:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:55:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:55:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:56:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:56:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:56:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:56:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:56:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:56:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:56:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:56:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 09:56:42:[cuda:0] Epoch 3000 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 09:56:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 09:56:48:mr: 3312.642 INFO:Trainer:2024-05-24 09:56:48:mrr: 0.148 INFO:Trainer:2024-05-24 09:56:48:hits@1: 0.115 INFO:Trainer:2024-05-24 09:56:48:hits@3: 0.16 INFO:Trainer:2024-05-24 09:56:48:hits@10: 0.211 INFO:Trainer:2024-05-24 09:56:48:mr_type: 333.072 INFO:Trainer:2024-05-24 09:56:48:mrr_type: 0.225 INFO:Trainer:2024-05-24 09:56:48:hits@1_type: 0.166 INFO:Trainer:2024-05-24 09:56:48:hits@3_type: 0.24 INFO:Trainer:2024-05-24 09:56:48:hits@10_type: 0.339 INFO:EarlyStopping:2024-05-24 09:56:48:Validation score improved (0.210000 --> 0.211000). Saving model ... INFO:Trainer:2024-05-24 09:56:48:[cuda:0] Epoch [3000/10000] | Batchsize: 1024 | loss: 0.331139 | 7.45703 seconds/epoch DEBUG:connectionpool:2024-05-24 09:56:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:56:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:57:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:57:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:57:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:57:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:57:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:57:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:57:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:57:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:57:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:57:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:57:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:57:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:58:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:58:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:58:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:58:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:58:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:58:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:58:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:58:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:58:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:58:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:58:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:59:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:59:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:59:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:59:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:59:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:59:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:59:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:59:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:59:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:59:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:59:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 09:59:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:00:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:00:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:00:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:00:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:00:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:00:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:00:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:00:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:00:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:00:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:00:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:01:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:01:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:01:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:01:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:01:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:01:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:01:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:01:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:01:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:01:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:01:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:01:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:02:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:02:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:02:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:02:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:02:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:02:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:02:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:02:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:02:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:02:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:02:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:03:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:03:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:03:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:03:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:03:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:03:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:03:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:03:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:03:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:03:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:03:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:04:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:04:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:04:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:04:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:04:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:04:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:04:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:04:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:04:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:04:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:04:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:04:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:05:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:05:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:05:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:05:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:05:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:05:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:05:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:05:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:05:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:05:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:05:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:06:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:06:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:06:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:06:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:06:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:06:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:06:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:06:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:06:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:06:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:06:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:06:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:07:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:07:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:07:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:07:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:07:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:07:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:07:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:07:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:07:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:07:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:08:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:08:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:08:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:08:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:08:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:08:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:08:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:08:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:08:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:08:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:08:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:09:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:09:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 10:09:10:[cuda:0] Epoch 3100 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 10:09:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 10:09:16:mr: 3272.665 INFO:Trainer:2024-05-24 10:09:16:mrr: 0.149 INFO:Trainer:2024-05-24 10:09:16:hits@1: 0.115 INFO:Trainer:2024-05-24 10:09:16:hits@3: 0.162 INFO:Trainer:2024-05-24 10:09:16:hits@10: 0.213 INFO:Trainer:2024-05-24 10:09:16:mr_type: 332.454 INFO:Trainer:2024-05-24 10:09:16:mrr_type: 0.226 INFO:Trainer:2024-05-24 10:09:16:hits@1_type: 0.167 INFO:Trainer:2024-05-24 10:09:16:hits@3_type: 0.241 INFO:Trainer:2024-05-24 10:09:16:hits@10_type: 0.34 INFO:EarlyStopping:2024-05-24 10:09:16:Validation score improved (0.211000 --> 0.213000). Saving model ... INFO:Trainer:2024-05-24 10:09:16:[cuda:0] Epoch [3100/10000] | Batchsize: 1024 | loss: 0.330365 | 7.45783 seconds/epoch DEBUG:connectionpool:2024-05-24 10:09:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:09:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:10:28:Starting new HTTPS connection (13): api.wandb.ai:443 DEBUG:connectionpool:2024-05-24 10:10:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:10:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:10:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:10:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:10:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:10:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:11:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:11:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:11:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:11:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:11:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:11:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:11:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:11:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:11:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:11:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:11:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:11:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:12:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:12:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:12:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:12:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:12:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:12:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:12:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:12:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:12:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:12:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:12:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:13:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:13:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:13:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:13:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:13:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:13:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:13:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:13:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:13:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:13:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:13:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:13:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:14:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:14:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:14:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:14:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:14:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:14:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:14:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:14:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:14:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:14:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:14:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:15:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:15:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:15:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:15:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:15:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:15:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:15:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:15:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:15:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:15:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:15:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:16:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:16:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:16:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:16:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:16:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:16:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:16:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:16:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:16:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:16:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:16:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:16:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:17:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:17:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:17:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:17:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:17:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:17:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:17:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:17:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:17:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:17:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:17:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:18:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:18:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:18:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:18:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:18:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:18:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:18:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:18:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:18:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:18:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:18:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:18:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:19:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:19:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:19:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:19:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:19:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:19:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:19:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:19:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:19:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:19:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:19:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:20:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:20:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:20:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:20:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:20:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:20:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:20:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:20:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:20:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:20:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:20:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:21:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:21:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:21:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:21:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:21:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:21:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 10:21:32:[cuda:0] Epoch 3200 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 10:21:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:21:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 10:21:38:mr: 3229.59 INFO:Trainer:2024-05-24 10:21:38:mrr: 0.15 INFO:Trainer:2024-05-24 10:21:38:hits@1: 0.116 INFO:Trainer:2024-05-24 10:21:38:hits@3: 0.163 INFO:Trainer:2024-05-24 10:21:38:hits@10: 0.214 INFO:Trainer:2024-05-24 10:21:38:mr_type: 331.879 INFO:Trainer:2024-05-24 10:21:38:mrr_type: 0.227 INFO:Trainer:2024-05-24 10:21:38:hits@1_type: 0.168 INFO:Trainer:2024-05-24 10:21:38:hits@3_type: 0.242 INFO:Trainer:2024-05-24 10:21:38:hits@10_type: 0.341 INFO:EarlyStopping:2024-05-24 10:21:38:Validation score improved (0.213000 --> 0.214000). Saving model ... INFO:Trainer:2024-05-24 10:21:38:[cuda:0] Epoch [3200/10000] | Batchsize: 1024 | loss: 0.329633 | 7.45657 seconds/epoch DEBUG:connectionpool:2024-05-24 10:21:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:21:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:21:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:21:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:22:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:22:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:22:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:22:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:22:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:22:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:22:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:22:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:22:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:22:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:22:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:23:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:23:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:23:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:23:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:23:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:23:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:23:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:23:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:23:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:23:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:23:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:23:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:24:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:24:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:24:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:24:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:24:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:24:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:24:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:24:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:24:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:24:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:24:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:25:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:25:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:25:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:25:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:25:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:25:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:25:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:25:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:25:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:25:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:25:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:26:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:26:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:26:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:26:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:26:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:26:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:26:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:26:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:26:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:26:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:26:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:26:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:27:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:27:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:27:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:27:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:27:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:27:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:27:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:27:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:27:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:27:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:27:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:28:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:28:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:28:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:28:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:28:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:28:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:28:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:28:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:28:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:28:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:28:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:29:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:29:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:29:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:29:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:29:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:29:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:29:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:29:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:29:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:29:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:29:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:29:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:30:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:30:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:30:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:30:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:30:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:30:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:30:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:30:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:30:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:30:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:30:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:31:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:31:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:31:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:31:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:31:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:31:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:31:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:31:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:31:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:31:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:31:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:31:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:32:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:32:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:32:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:32:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:32:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:32:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:32:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:32:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:32:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:32:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:32:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:33:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:33:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:33:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:33:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:33:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:33:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:33:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:33:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:33:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:33:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 10:33:54:[cuda:0] Epoch 3300 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 10:33:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:34:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 10:34:00:mr: 3186.011 INFO:Trainer:2024-05-24 10:34:00:mrr: 0.151 INFO:Trainer:2024-05-24 10:34:00:hits@1: 0.117 INFO:Trainer:2024-05-24 10:34:00:hits@3: 0.163 INFO:Trainer:2024-05-24 10:34:00:hits@10: 0.216 INFO:Trainer:2024-05-24 10:34:00:mr_type: 331.419 INFO:Trainer:2024-05-24 10:34:00:mrr_type: 0.228 INFO:Trainer:2024-05-24 10:34:00:hits@1_type: 0.168 INFO:Trainer:2024-05-24 10:34:00:hits@3_type: 0.242 INFO:Trainer:2024-05-24 10:34:00:hits@10_type: 0.342 INFO:EarlyStopping:2024-05-24 10:34:00:Validation score improved (0.214000 --> 0.216000). Saving model ... INFO:Trainer:2024-05-24 10:34:00:[cuda:0] Epoch [3300/10000] | Batchsize: 1024 | loss: 0.329105 | 7.45540 seconds/epoch DEBUG:connectionpool:2024-05-24 10:34:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:34:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:34:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:34:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:34:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:34:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:34:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:34:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:34:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:34:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:34:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:35:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:35:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:35:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:35:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:35:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:35:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:35:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:35:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:35:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:35:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:35:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:36:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:36:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:36:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:36:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:36:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:36:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:36:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:36:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:36:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:36:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:36:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:36:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:37:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:37:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:37:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:37:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:37:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:37:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:37:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:37:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:37:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:37:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:37:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:38:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:38:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:38:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:38:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:38:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:38:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:38:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:38:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:38:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:38:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:38:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:39:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:39:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:39:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:39:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:39:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:39:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:39:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:39:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:39:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:39:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:39:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:39:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:40:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:40:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:40:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:40:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:40:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:40:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:40:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:40:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:40:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:40:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:40:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:41:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:41:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:41:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:41:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:41:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:41:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:41:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:41:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:41:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:41:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:41:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:41:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:42:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:42:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:42:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:42:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:42:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:42:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:42:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:42:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:42:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:42:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:42:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:43:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:43:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:43:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:43:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:43:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:43:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:43:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:43:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:43:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:43:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:43:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:44:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:44:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:44:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:44:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:44:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:44:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:44:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:44:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:44:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:44:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:44:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:44:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:45:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:45:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:45:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:45:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:45:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:45:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:45:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:45:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:45:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:45:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:45:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:46:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:46:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:46:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 10:46:15:[cuda:0] Epoch 3400 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 10:46:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 10:46:21:mr: 3169.373 INFO:Trainer:2024-05-24 10:46:21:mrr: 0.152 INFO:Trainer:2024-05-24 10:46:21:hits@1: 0.118 INFO:Trainer:2024-05-24 10:46:21:hits@3: 0.163 INFO:Trainer:2024-05-24 10:46:21:hits@10: 0.217 INFO:Trainer:2024-05-24 10:46:21:mr_type: 331.113 INFO:Trainer:2024-05-24 10:46:21:mrr_type: 0.228 INFO:Trainer:2024-05-24 10:46:21:hits@1_type: 0.168 INFO:Trainer:2024-05-24 10:46:21:hits@3_type: 0.242 INFO:Trainer:2024-05-24 10:46:21:hits@10_type: 0.342 INFO:EarlyStopping:2024-05-24 10:46:21:Validation score improved (0.216000 --> 0.217000). Saving model ... INFO:Trainer:2024-05-24 10:46:21:[cuda:0] Epoch [3400/10000] | Batchsize: 1024 | loss: 0.328428 | 7.45435 seconds/epoch DEBUG:connectionpool:2024-05-24 10:46:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:46:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:46:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:46:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:46:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:46:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:46:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:47:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:47:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:47:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:47:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:47:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:47:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:47:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:47:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:47:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:47:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:47:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:47:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:48:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:48:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:48:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:48:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:48:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:48:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:48:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:48:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:48:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:48:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:48:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:49:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:49:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:49:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:49:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:49:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:49:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:49:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:49:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:49:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:49:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:49:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:49:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:50:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:50:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:50:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:50:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:50:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:50:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:50:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:50:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:50:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:50:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:50:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:51:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:51:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:51:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:51:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:51:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:51:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:51:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:51:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:51:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:51:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:51:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:52:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:52:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:52:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:52:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:52:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:52:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:52:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:52:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:52:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:52:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:52:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:52:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:53:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:53:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:53:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:53:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:53:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:53:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:53:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:53:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:53:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:53:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:53:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:54:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:54:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:54:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:54:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:54:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:54:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:54:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:54:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:54:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:54:50:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:54:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:55:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:55:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:55:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:55:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:55:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:55:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:55:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:55:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:55:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:55:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:55:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:55:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:56:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:56:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:56:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:56:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:56:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:56:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:56:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:56:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:56:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:56:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:56:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:57:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:57:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:57:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:57:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:57:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:57:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:57:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:57:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:57:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:57:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:57:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:57:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:58:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:58:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:58:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:58:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:58:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:58:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:58:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 10:58:38:[cuda:0] Epoch 3500 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 10:58:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 10:58:44:mr: 3166.035 INFO:Trainer:2024-05-24 10:58:44:mrr: 0.152 INFO:Trainer:2024-05-24 10:58:44:hits@1: 0.118 INFO:Trainer:2024-05-24 10:58:44:hits@3: 0.163 INFO:Trainer:2024-05-24 10:58:44:hits@10: 0.217 INFO:Trainer:2024-05-24 10:58:44:mr_type: 330.936 INFO:Trainer:2024-05-24 10:58:44:mrr_type: 0.228 INFO:Trainer:2024-05-24 10:58:44:hits@1_type: 0.168 INFO:Trainer:2024-05-24 10:58:44:hits@3_type: 0.242 INFO:Trainer:2024-05-24 10:58:44:hits@10_type: 0.342 INFO:EarlyStopping:2024-05-24 10:58:44:EarlyStopping counter: 1 / 2 INFO:Trainer:2024-05-24 10:58:44:[cuda:0] Epoch [3500/10000] | Batchsize: 1024 | loss: 0.328385 | 7.45355 seconds/epoch DEBUG:connectionpool:2024-05-24 10:58:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:58:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:58:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:59:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:59:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:59:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:59:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:59:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:59:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:59:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:59:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:59:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:59:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 10:59:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:00:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:00:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:00:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:00:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:00:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:00:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:00:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:00:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:00:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:00:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:00:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:00:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:01:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:01:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:01:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:01:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:01:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:01:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:01:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:01:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:01:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:01:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:01:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:02:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:02:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:02:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:02:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:02:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:02:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:02:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:02:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:02:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:02:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:02:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:02:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:03:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:03:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:03:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:03:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:03:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:03:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:03:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:03:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:03:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:03:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:03:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:04:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:04:08:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:04:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:04:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:04:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:04:29:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:04:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:04:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:04:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:04:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:04:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:05:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:05:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:05:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:05:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:05:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:05:27:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:05:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:05:38:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:05:43:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:05:48:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:05:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:05:59:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:06:04:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:06:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:06:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:06:20:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:06:25:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:06:31:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:06:36:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:06:41:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:06:46:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:06:52:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:06:57:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:07:02:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:07:07:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:07:13:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:07:18:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:07:23:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:07:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:07:34:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:07:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:07:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:07:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:07:55:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:08:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:08:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:08:11:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:08:16:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:08:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:08:26:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:08:32:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:08:37:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:08:42:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:08:47:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:08:53:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:08:58:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:09:03:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:09:09:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:09:14:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:09:19:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:09:24:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:09:30:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:09:35:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:09:40:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:09:45:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:09:51:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:09:56:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:10:01:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:10:06:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:10:12:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:10:17:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:10:22:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:10:28:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:10:33:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:10:38:Resetting dropped connection: api.wandb.ai DEBUG:connectionpool:2024-05-24 11:10:39:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:10:44:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:10:49:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:10:54:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:11:00:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 11:11:01:[cuda:0] Epoch 3600 | The model starts evaluation on the validation set. DEBUG:connectionpool:2024-05-24 11:11:05:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 11:11:07:mr: 3161.462 INFO:Trainer:2024-05-24 11:11:07:mrr: 0.152 INFO:Trainer:2024-05-24 11:11:07:hits@1: 0.118 INFO:Trainer:2024-05-24 11:11:07:hits@3: 0.163 INFO:Trainer:2024-05-24 11:11:07:hits@10: 0.217 INFO:Trainer:2024-05-24 11:11:07:mr_type: 330.917 INFO:Trainer:2024-05-24 11:11:07:mrr_type: 0.228 INFO:Trainer:2024-05-24 11:11:07:hits@1_type: 0.168 INFO:Trainer:2024-05-24 11:11:07:hits@3_type: 0.242 INFO:Trainer:2024-05-24 11:11:07:hits@10_type: 0.342 INFO:EarlyStopping:2024-05-24 11:11:07:EarlyStopping counter: 2 / 2 INFO:Trainer:2024-05-24 11:11:07:[cuda:0] Send an early stopping signal INFO:Trainer:2024-05-24 11:11:07:[cuda:0] The model training is completed, taking a total of 26829.96256 seconds. INFO:Trainer:2024-05-24 11:11:07:[cuda:0] Model saved at ./model.pth. INFO:Trainer:2024-05-24 11:11:07:[cuda:0] The model starts evaluating in the test set. DEBUG:connectionpool:2024-05-24 11:11:10:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None DEBUG:connectionpool:2024-05-24 11:11:15:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None INFO:Trainer:2024-05-24 11:11:16:mr: 3153.072 INFO:Trainer:2024-05-24 11:11:16:mrr: 0.151 INFO:Trainer:2024-05-24 11:11:16:hits@1: 0.117 INFO:Trainer:2024-05-24 11:11:16:hits@3: 0.163 INFO:Trainer:2024-05-24 11:11:16:hits@10: 0.215 INFO:Trainer:2024-05-24 11:11:16:mr_type: 332.005 INFO:Trainer:2024-05-24 11:11:16:mrr_type: 0.226 INFO:Trainer:2024-05-24 11:11:16:hits@1_type: 0.167 INFO:Trainer:2024-05-24 11:11:16:hits@3_type: 0.24 INFO:Trainer:2024-05-24 11:11:16:hits@10_type: 0.338 wandb: - 0.014 MB of 0.014 MB uploaded wandb: \ 0.014 MB of 0.014 MB uploaded wandb: | 0.014 MB of 0.547 MB uploaded DEBUG:connectionpool:2024-05-24 11:11:21:https://api.wandb.ai:443 "POST /graphql HTTP/1.1" 200 None wandb: / 0.018 MB of 0.547 MB uploaded wandb: - 0.547 MB of 0.547 MB uploaded wandb: \ 0.547 MB of 0.547 MB uploaded wandb: | 0.547 MB of 0.547 MB uploaded wandb: wandb: wandb: Run history: wandb: duration โ– wandb: test/hits@1 โ– wandb: test/hits@10 โ– wandb: test/hits@10_type โ– wandb: test/hits@1_type โ– wandb: test/hits@3 โ– wandb: test/hits@3_type โ– wandb: test/mr โ– wandb: test/mr_type โ– wandb: test/mrr โ– wandb: test/mrr_type โ– wandb: train/epoch โ–โ–โ–โ–‚โ–‚โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆ wandb: train/train_loss โ–ˆโ–†โ–…โ–„โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ–โ–โ–โ– wandb: val/epoch โ–โ–โ–โ–‚โ–‚โ–‚โ–‚โ–‚โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆ wandb: val/hits@1 โ–โ–„โ–„โ–„โ–…โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@10 โ–โ–‚โ–ƒโ–ƒโ–„โ–„โ–…โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@10_type โ–โ–‚โ–ƒโ–„โ–„โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@1_type โ–โ–ƒโ–ƒโ–„โ–„โ–…โ–…โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@3 โ–โ–ƒโ–ƒโ–„โ–„โ–„โ–„โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/hits@3_type โ–โ–‚โ–ƒโ–„โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/mr โ–ˆโ–‡โ–†โ–†โ–†โ–…โ–…โ–„โ–„โ–„โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ– wandb: val/mr_type โ–ˆโ–‡โ–‡โ–†โ–†โ–†โ–…โ–…โ–…โ–…โ–„โ–„โ–„โ–„โ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–ƒโ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–‚โ–โ–โ–โ–โ–โ–โ–โ– wandb: val/mrr โ–โ–ƒโ–ƒโ–„โ–„โ–„โ–…โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: val/mrr_type โ–โ–‚โ–ƒโ–„โ–„โ–…โ–…โ–†โ–†โ–†โ–†โ–†โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–‡โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ wandb: wandb: Run summary: wandb: duration 26829.96256 wandb: test/hits@1 0.117 wandb: test/hits@10 0.215 wandb: test/hits@10_type 0.338 wandb: test/hits@1_type 0.167 wandb: test/hits@3 0.163 wandb: test/hits@3_type 0.24 wandb: test/mr 3153.072 wandb: test/mr_type 332.005 wandb: test/mrr 0.151 wandb: test/mrr_type 0.226 wandb: train/epoch 3500 wandb: train/train_loss 0.32838 wandb: val/epoch 3599 wandb: val/hits@1 0.118 wandb: val/hits@10 0.217 wandb: val/hits@10_type 0.342 wandb: val/hits@1_type 0.168 wandb: val/hits@3 0.163 wandb: val/hits@3_type 0.242 wandb: val/mr 3161.462 wandb: val/mr_type 330.917 wandb: val/mrr 0.152 wandb: val/mrr_type 0.228 wandb: wandb: ๐Ÿš€ View run glamorous-sweep-3 at: https://wandb.ai/3555028709/pybind11-ke-sweeps/runs/wsmfz2ez wandb: โญ๏ธ View project at: https://wandb.ai/3555028709/pybind11-ke-sweeps wandb: Synced 5 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s) wandb: Find logs at: ./wandb/run-20240524_034344-wsmfz2ez/logs DEBUG:connectionpool:2024-05-24 11:11:24:Resetting dropped connection: o151352.ingest.sentry.io DEBUG:connectionpool:2024-05-24 11:11:24:https://o151352.ingest.sentry.io:443 "POST /api/4504800232407040/envelope/ HTTP/1.1" 200 0