Final Phase error log request ¶
By: ehrm0rh on March 1, 2022, 12:37 p.m.
Hi,
I'm a team member of YC_LEE. I'm trying to submit the final phase model, but I met error...
Can I get the stderr for latest submissions?
Thank you.
Hyunbin cho
Final Phase error log request
Re: Final Phase error log request ¶
By: coendevente on March 1, 2022, 2:13 p.m.
Hi,
This is the stderr of one of the runs from your failed submission:
2022-03-01T12:26:25+00:00 2022-03-01 21:26:25.464969: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-03-01T12:26:25+00:00 2022-03-01 21:26:25.474367: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-03-01T12:26:25+00:00 2022-03-01 21:26:25.475024: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-03-01T12:26:25+00:00 2022-03-01 21:26:25.475864: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
2022-03-01T12:26:25+00:00 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-03-01T12:26:25+00:00 2022-03-01 21:26:25.476583: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-03-01T12:26:25+00:00 2022-03-01 21:26:25.477231: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-03-01T12:26:25+00:00 2022-03-01 21:26:25.477789: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-03-01T12:26:26+00:00 2022-03-01 21:26:26.067444: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-03-01T12:26:26+00:00 2022-03-01 21:26:26.068119: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-03-01T12:26:26+00:00 2022-03-01 21:26:26.068722: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:936] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-03-01T12:26:26+00:00 2022-03-01 21:26:26.069267: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 13797 MB memory: -> device: 0, name: Tesla T4, pci bus id: 0000:00:1e.0, compute capability: 7.5
2022-03-01T12:28:43+00:00
0%| | 0/300 [00:00<?, ?it/s]2022-03-01 21:28:43.907631: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8204
2022-03-01T12:28:44+00:00 2022-03-01 21:28:44.590214: I tensorflow/core/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
2022-03-01T12:28:44+00:00 2022-03-01 21:28:44.590672: I tensorflow/core/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
2022-03-01T12:28:44+00:00 2022-03-01 21:28:44.590693: W tensorflow/stream_executor/gpu/asm_compiler.cc:80] Couldn't get ptxas version string: INTERNAL: Couldn't invoke ptxas --version
2022-03-01T12:28:44+00:00 2022-03-01 21:28:44.591148: I tensorflow/core/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
2022-03-01T12:28:44+00:00 2022-03-01 21:28:44.591207: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] INTERNAL: Failed to launch ptxas
2022-03-01T12:28:44+00:00 Relying on driver to perform ptx compilation.
2022-03-01T12:28:44+00:00 Modify $PATH to customize ptxas location.
2022-03-01T12:28:44+00:00 This message will be only logged once.
2022-03-01T12:42:07+00:00
0%| | 1/300 [00:10<50:55, 10.22s/it]
1%| | 2/300 [00:14<32:48, 6.60s/it]
1%| | 3/300 [00:16<22:44, 4.60s/it]
1%|▏ | 4/300 [00:19<18:50, 3.82s/it]
2%|▏ | 5/300 [00:21<16:23, 3.33s/it]
2%|▏ | 6/300 [00:23<14:25, 2.94s/it]
2%|▏ | 7/300 [00:30<20:23, 4.18s/it]
3%|▎ | 8/300 [00:33<18:59, 3.90s/it]
3%|▎ | 9/300 [00:39<21:45, 4.49s/it]
3%|▎ | 10/300 [00:43<21:08, 4.37s/it]
4%|▎ | 11/300 [00:45<17:23, 3.61s/it]
4%|▍ | 12/300 [00:47<14:52, 3.10s/it]
4%|▍ | 13/300 [00:50<13:58, 2.92s/it]
5%|▍ | 14/300 [00:52<13:40, 2.87s/it]
5%|▌ | 15/300 [00:57<16:54, 3.56s/it]
5%|▌ | 16/300 [01:00<15:25, 3.26s/it]
6%|▌ | 17/300 [01:02<13:40, 2.90s/it]
6%|▌ | 18/300 [01:05<14:15, 3.03s/it]
6%|▋ | 19/300 [01:09<14:33, 3.11s/it]
7%|▋ | 20/300 [01:15<19:03, 4.09s/it]
7%|▋ | 21/300 [01:18<17:17, 3.72s/it]
7%|▋ | 22/300 [01:23<19:01, 4.10s/it]
8%|▊ | 23/300 [01:26<17:48, 3.86s/it]
8%|▊ | 24/300 [01:28<14:38, 3.18s/it]
8%|▊ | 25/300 [01:33<17:06, 3.73s/it]
9%|▊ | 26/300 [01:39<20:36, 4.51s/it]
9%|▉ | 27/300 [01:42<18:03, 3.97s/it]
9%|▉ | 28/300 [01:45<16:16, 3.59s/it]
10%|▉ | 29/300 [01:47<14:48, 3.28s/it]
10%|█ | 30/300 [01:53<18:52, 4.20s/it]
10%|█ | 31/300 [01:58<18:49, 4.20s/it]
11%|█ | 32/300 [02:01<16:59, 3.80s/it]
11%|█ | 33/300 [02:06<18:46, 4.22s/it]
11%|█▏ | 34/300 [02:08<16:42, 3.77s/it]
12%|█▏ | 35/300 [02:11<15:08, 3.43s/it]
12%|█▏ | 36/300 [02:14<14:05, 3.20s/it]
12%|█▏ | 37/300 [02:20<18:31, 4.23s/it]
13%|█▎ | 38/300 [02:23<16:38, 3.81s/it]
13%|█▎ | 39/300 [02:28<17:20, 3.99s/it]
13%|█▎ | 40/300 [02:30<15:26, 3.56s/it]
14%|█▎ | 41/300 [02:33<14:39, 3.39s/it]
14%|█▍ | 42/300 [02:37<15:39, 3.64s/it]
14%|█▍ | 43/300 [02:42<16:13, 3.79s/it]
15%|█▍ | 44/300 [02:44<14:51, 3.48s/it]
15%|█▌ | 45/300 [02:51<18:32, 4.36s/it]
15%|█▌ | 46/300 [02:55<17:49, 4.21s/it]
16%|█▌ | 47/300 [02:57<15:44, 3.73s/it]
16%|█▌ | 48/300 [03:01<16:14, 3.87s/it]
16%|█▋ | 49/300 [03:03<13:34, 3.24s/it]
17%|█▋ | 50/300 [03:06<12:59, 3.12s/it]
17%|█▋ | 51/300 [03:11<15:13, 3.67s/it]
17%|█▋ | 52/300 [03:13<13:19, 3.22s/it]
18%|█▊ | 53/300 [03:18<15:23, 3.74s/it]
18%|█▊ | 54/300 [03:23<16:55, 4.13s/it]
18%|█▊ | 55/300 [03:26<15:18, 3.75s/it]
19%|█▊ | 56/300 [03:28<12:58, 3.19s/it]
19%|█▉ | 57/300 [03:32<13:41, 3.38s/it]
19%|█▉ | 58/300 [03:34<12:10, 3.02s/it]
20%|█▉ | 59/300 [03:38<13:27, 3.35s/it]
20%|██ | 60/300 [03:40<12:23, 3.10s/it]
20%|██ | 61/300 [03:43<12:03, 3.03s/it]
21%|██ | 62/300 [03:45<10:28, 2.64s/it]
21%|██ | 63/300 [03:51<14:48, 3.75s/it]
21%|██▏ | 64/300 [03:53<12:20, 3.14s/it]
22%|██▏ | 65/300 [03:55<10:39, 2.72s/it]
22%|██▏ | 66/300 [03:58<10:31, 2.70s/it]
22%|██▏ | 67/300 [04:02<12:04, 3.11s/it]
23%|██▎ | 68/300 [04:04<11:31, 2.98s/it]
23%|██▎ | 69/300 [04:08<12:50, 3.34s/it]
23%|██▎ | 70/300 [04:10<10:47, 2.81s/it]
24%|██▎ | 71/300 [04:14<12:13, 3.20s/it]
24%|██▍ | 72/300 [04:17<11:40, 3.07s/it]
24%|██▍ | 73/300 [04:21<12:49, 3.39s/it]
25%|██▍ | 74/300 [04:23<11:23, 3.02s/it]
25%|██▌ | 75/300 [04:28<13:27, 3.59s/it]
25%|██▌ | 76/300 [04:31<12:03, 3.23s/it]
26%|██▌ | 77/300 [04:40<18:55, 5.09s/it]
26%|██▌ | 78/300 [04:46<20:22, 5.51s/it]
26%|██▋ | 79/300 [04:49<17:20, 4.71s/it]
27%|██▋ | 80/300 [04:52<14:34, 3.98s/it]
27%|██▋ | 81/300 [04:56<14:40, 4.02s/it]
27%|██▋ | 82/300 [04:58<13:10, 3.63s/it]
28%|██▊ | 83/300 [05:01<12:10, 3.37s/it]
28%|██▊ | 84/300 [05:03<10:41, 2.97s/it]
28%|██▊ | 85/300 [05:08<12:06, 3.38s/it]
29%|██▊ | 86/300 [05:10<11:07, 3.12s/it]
29%|██▉ | 87/300 [05:16<13:58, 3.94s/it]
29%|██▉ | 88/300 [05:17<11:18, 3.20s/it]
30%|██▉ | 89/300 [05:19<09:46, 2.78s/it]
30%|███ | 90/300 [05:21<08:21, 2.39s/it]
30%|███ | 91/300 [05:27<12:26, 3.57s/it]
31%|███ | 92/300 [05:31<13:02, 3.76s/it]
31%|███ | 93/300 [05:36<14:15, 4.13s/it]
31%|███▏ | 94/300 [05:41<15:04, 4.39s/it]
32%|███▏ | 95/300 [05:45<14:03, 4.11s/it]
32%|███▏ | 96/300 [05:51<16:14, 4.78s/it]
32%|███▏ | 97/300 [05:54<14:13, 4.20s/it]
33%|███▎ | 98/300 [05:58<13:46, 4.09s/it]
33%|███▎ | 99/300 [06:03<14:37, 4.36s/it]
33%|███▎ | 100/300 [06:05<12:38, 3.79s/it]
34%|███▎ | 101/300 [06:11<15:07, 4.56s/it]
34%|███▍ | 102/300 [06:13<12:16, 3.72s/it]
34%|███▍ | 103/300 [06:16<11:02, 3.37s/it]
35%|███▍ | 104/300 [06:18<09:29, 2.91s/it]
35%|███▌ | 105/300 [06:21<09:51, 3.04s/it]
35%|███▌ | 106/300 [06:23<08:54, 2.75s/it]
36%|███▌ | 107/300 [06:28<11:03, 3.44s/it]
36%|███▌ | 108/300 [06:31<10:26, 3.26s/it]
36%|███▋ | 109/300 [06:37<13:22, 4.20s/it]
37%|███▋ | 110/300 [06:41<12:27, 3.94s/it]
37%|███▋ | 111/300 [06:47<14:39, 4.66s/it]
37%|███▋ | 112/300 [06:48<11:38, 3.72s/it]
38%|███▊ | 113/300 [06:50<09:50, 3.16s/it]
38%|███▊ | 114/300 [06:52<08:29, 2.74s/it]
38%|███▊ | 115/300 [06:56<09:27, 3.07s/it]
39%|███▊ | 116/300 [06:59<09:08, 2.98s/it]
39%|███▉ | 117/300 [07:02<08:59, 2.95s/it]
39%|███▉ | 118/300 [07:04<08:36, 2.84s/it]
40%|███▉ | 119/300 [07:08<09:41, 3.21s/it]
40%|████ | 120/300 [07:11<08:56, 2.98s/it]
40%|████ | 121/300 [07:14<08:45, 2.94s/it]
41%|████ | 122/300 [07:16<08:26, 2.85s/it]
41%|████ | 123/300 [07:21<10:32, 3.57s/it]
41%|████▏ | 124/300 [07:24<09:52, 3.36s/it]
42%|████▏ | 125/300 [07:29<10:37, 3.64s/it]
42%|████▏ | 126/300 [07:34<11:51, 4.09s/it]
42%|████▏ | 127/300 [07:39<12:41, 4.40s/it]
43%|████▎ | 128/300 [07:42<11:13, 3.92s/it]
43%|████▎ | 129/300 [07:48<13:30, 4.74s/it]
43%|████▎ | 130/300 [07:51<11:26, 4.04s/it]
44%|████▎ | 131/300 [07:54<10:23, 3.69s/it]
44%|████▍ | 132/300 [07:56<09:20, 3.34s/it]
44%|████▍ | 133/300 [08:00<09:56, 3.57s/it]
45%|████▍ | 134/300 [08:03<09:11, 3.32s/it]
45%|████▌ | 135/300 [08:08<10:42, 3.89s/it]
45%|████▌ | 136/300 [08:15<12:40, 4.64s/it]
46%|████▌ | 137/300 [08:19<12:44, 4.69s/it]
46%|████▌ | 138/300 [08:21<10:31, 3.90s/it]
46%|████▋ | 139/300 [08:23<08:44, 3.26s/it]
47%|████▋ | 140/300 [08:25<07:29, 2.81s/it]
47%|████▋ | 141/300 [08:27<06:39, 2.51s/it]
47%|████▋ | 142/300 [08:30<07:14, 2.75s/it]
48%|████▊ | 143/300 [08:36<09:34, 3.66s/it]
48%|████▊ | 144/300 [08:39<08:44, 3.36s/it]
48%|████▊ | 145/300 [08:44<10:02, 3.89s/it]
49%|████▊ | 146/300 [08:46<09:11, 3.58s/it]
49%|████▉ | 147/300 [08:53<11:14, 4.41s/it]
49%|████▉ | 148/300 [08:58<11:45, 4.64s/it]
50%|████▉ | 149/300 [09:03<11:55, 4.74s/it]
50%|█████ | 150/300 [09:05<10:10, 4.07s/it]
50%|█████ | 151/300 [09:07<08:14, 3.32s/it]
51%|█████ | 152/300 [09:13<10:26, 4.24s/it]
51%|█████ | 153/300 [09:16<09:22, 3.83s/it]
51%|█████▏ | 154/300 [09:20<08:58, 3.69s/it]
52%|█████▏ | 155/300 [09:24<09:10, 3.80s/it]
52%|█████▏ | 156/300 [09:27<08:44, 3.64s/it]
52%|█████▏ | 157/300 [09:30<07:59, 3.35s/it]
53%|█████▎ | 158/300 [09:36<10:01, 4.23s/it]
53%|█████▎ | 159/300 [09:38<08:33, 3.64s/it]
53%|█████▎ | 160/300 [09:41<08:12, 3.52s/it]
54%|█████▎ | 161/300 [09:47<09:43, 4.20s/it]
54%|█████▍ | 162/300 [09:50<08:41, 3.78s/it]
54%|█████▍ | 163/300 [09:52<07:05, 3.10s/it]
55%|█████▍ | 164/300 [09:53<05:55, 2.61s/it]
55%|█████▌ | 165/300 [09:59<07:50, 3.48s/it]
55%|█████▌ | 166/300 [10:03<08:17, 3.71s/it]
56%|█████▌ | 167/300 [10:05<07:12, 3.25s/it]
56%|█████▌ | 168/300 [10:07<06:28, 2.94s/it]
56%|█████▋ | 169/300 [10:10<06:11, 2.83s/it]
57%|█████▋ | 170/300 [10:13<06:15, 2.89s/it]
57%|█████▋ | 171/300 [10:15<05:41, 2.65s/it]
57%|█████▋ | 172/300 [10:18<05:45, 2.70s/it]
58%|█████▊ | 173/300 [10:22<06:37, 3.13s/it]
58%|█████▊ | 174/300 [10:26<06:59, 3.33s/it]
58%|█████▊ | 175/300 [10:30<07:24, 3.56s/it]
59%|█████▊ | 176/300 [10:36<09:06, 4.40s/it]
59%|█████▉ | 177/300 [10:39<07:50, 3.83s/it]
59%|█████▉ | 178/300 [10:44<08:57, 4.40s/it]
60%|█████▉ | 179/300 [10:48<08:40, 4.30s/it]
60%|██████ | 180/300 [10:51<07:19, 3.67s/it]
60%|██████ | 181/300 [10:53<06:22, 3.22s/it]
61%|██████ | 182/300 [10:55<05:53, 2.99s/it]
61%|██████ | 183/300 [10:57<05:18, 2.72s/it]
61%|██████▏ | 184/300 [11:02<06:40, 3.45s/it]
62%|██████▏ | 185/300 [11:07<07:03, 3.69s/it]
62%|██████▏ | 186/300 [11:12<07:49, 4.12s/it]
62%|██████▏ | 187/300 [11:18<09:01, 4.79s/it]
63%|██████▎ | 188/300 [11:26<10:49, 5.80s/it]
63%|██████▎ | 189/300 [11:29<09:06, 4.92s/it]
63%|██████▎ | 190/300 [11:31<07:25, 4.05s/it]
64%|██████▎ | 191/300 [11:35<07:01, 3.87s/it]
64%|██████▍ | 192/300 [11:36<05:48, 3.23s/it]
64%|██████▍ | 193/300 [11:39<05:11, 2.92s/it]
65%|██████▍ | 194/300 [11:41<05:00, 2.84s/it]
65%|██████▌ | 195/300 [11:44<04:58, 2.84s/it]
65%|██████▌ | 196/300 [11:47<04:56, 2.85s/it]
66%|██████▌ | 197/300 [11:53<06:24, 3.73s/it]
66%|██████▌ | 198/300 [11:59<07:26, 4.37s/it]
66%|██████▋ | 199/300 [12:03<07:11, 4.27s/it]
67%|██████▋ | 200/300 [12:04<05:43, 3.44s/it]
67%|██████▋ | 201/300 [12:07<05:12, 3.15s/it]
67%|██████▋ | 202/300 [12:13<06:40, 4.09s/it]
68%|██████▊ | 203/300 [12:17<06:34, 4.07s/it]
68%|██████▊ | 204/300 [12:23<07:29, 4.68s/it]
68%|██████▊ | 205/300 [12:27<07:07, 4.50s/it]
69%|██████▊ | 206/300 [12:30<06:29, 4.14s/it]
69%|██████▉ | 207/300 [12:36<06:52, 4.43s/it]
69%|██████▉ | 208/300 [12:37<05:25, 3.54s/it]
70%|██████▉ | 209/300 [12:42<06:12, 4.09s/it]
70%|███████ | 210/300 [12:45<05:23, 3.60s/it]
70%|███████ | 211/300 [12:48<04:58, 3.35s/it]
71%|███████ | 212/300 [12:50<04:34, 3.12s/it]
71%|███████ | 213/300 [12:53<04:19, 2.99s/it]
71%|███████▏ | 214/300 [12:55<04:07, 2.88s/it]
72%|███████▏ | 215/300 [12:58<03:46, 2.67s/it]
72%|███████▏ | 216/300 [13:02<04:21, 3.11s/it]
72%|███████▏ | 217/300 [13:04<04:05, 2.96s/it]
73%|███████▎ | 218/300 [13:07<03:54, 2.86s/it]
73%|███████▎ | 219/300 [13:12<04:46, 3.54s/it]
73%|███████▎ | 220/300 [13:19<05:51, 4.39s/it]
74%|███████▎ | 221/300 [13:21<05:11, 3.94s/it]
74%|███████▍ | 222/300 [13:24<04:32, 3.49s/it]
74%|███████▍ | 223/300 [13:26<03:58, 3.10s/it]
75%|███████▍ | 224/300 [13:27<03:17, 2.60s/it]
75%|███████▍ | 224/300 [13:28<04:34, 3.61s/it]
2022-03-01T12:42:07+00:00 Traceback (most recent call last):
2022-03-01T12:42:07+00:00 File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
2022-03-01T12:42:07+00:00 return _run_code(code, main_globals, None,
2022-03-01T12:42:07+00:00 File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
2022-03-01T12:42:07+00:00 exec(code, run_globals)
2022-03-01T12:42:07+00:00 File "/opt/algorithm/process.py", line 193, in <module>
2022-03-01T12:42:07+00:00 airogs_algorithm().process()
2022-03-01T12:42:07+00:00 File "/home/algorithm/.local/lib/python3.8/site-packages/evalutils/evalutils.py", line 183, in process
2022-03-01T12:42:07+00:00 self.process_cases()
2022-03-01T12:42:07+00:00 File "/home/algorithm/.local/lib/python3.8/site-packages/evalutils/evalutils.py", line 191, in process_cases
2022-03-01T12:42:07+00:00 self._case_results.append(self.process_case(idx=idx, case=case))
2022-03-01T12:42:07+00:00 File "/opt/algorithm/process.py", line 109, in process_case
2022-03-01T12:42:07+00:00 results.append(self.predict(input_image_array=input_image_array))
2022-03-01T12:42:07+00:00 File "/opt/algorithm/process.py", line 128, in predict
2022-03-01T12:42:07+00:00 utils.img_resizing(
2022-03-01T12:42:07+00:00 File "/opt/algorithm/utils.py", line 224, in img_resizing
2022-03-01T12:42:07+00:00 return cv2.resize(img, (608, 608), interpolation=cv2.INTER_NEAREST)
2022-03-01T12:42:07+00:00 cv2.error: OpenCV(4.5.5) /io/opencv/modules/imgproc/src/resize.cpp:4052: error: (-215:Assertion failed) !ssize.empty() in function 'resize'
2022-03-01T12:42:07+00:00
2022-03-01T12:42:08+00:00 WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To silence these warnings, use 'status.expect_partial()'. See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the restore function.
2022-03-01T12:42:08+00:00 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.decay
2022-03-01T12:42:08+00:00 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.learning_rate
2022-03-01T12:42:08+00:00 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.momentum
2022-03-01T12:42:08+00:00 WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.iter
Best regards, Coen de Vente
