Modifications for testing

• train_data -> modify that this script does the training (no testing)
• test_training -> modify this for test purposes
• reinitialize path for pytest
• fix bug within training!

assigned to @nik20652

created branch 45-modifications-for-testing to address this issue

marked the checklist item train_data -> modify that this script does the training (no testing) as completed

marked the checklist item test_training -> modify this for test purposes as completed

changed the description

marked the checklist item reinitialize path for pytest as completed

Hey @hanikevi ,

unfortunately we got a bug within training.

The problem is that after we run for batch in batches in the first epoch we only get nan as np.mean(batch_loss_array). It looks like there is no training after this first iteration. It can be reproduced in this branch running python -m run.tests.test_training for example. I added an assertion statement in trainer that returns an error when nan occurs for the first time.

Maybe you have an idea what might be the reason for this! Thanks anyway!

You mean after the first epoch? I'd check that you're resetting the iterator after an epoch. IIRC we init the iterator when we init a dataloader, and the iterator is a list of batch idxs, so it'll run out. I think the pytorch dataloaders have a reset function, I forget if we have something similar already written

yes exactly. Ok nice, I'll check that.

Uff very nice suggestion! I bumped DynSliceDataloader and modified the initialization of the sampler. Every time the __iter__() is called, the sampler gets reinitialized. Now the iter() looks like:

def __iter__(self) -> Iterable[Batch]:
        self._sampler = iter(self._get_idx_batch())
        return self

For me it seems to work! Thanks!

marked the checklist item fix bug within training! as completed

mentioned in merge request !31 (merged)

mentioned in commit 83f4c83b

closed with merge request !31 (merged)

mentioned in issue #44 (closed)