Skip to content

Apply new code structure

Lisa-Marie Fenner requested to merge 24-structure-the-code into main

There are many changes in #24 (closed) I like to discuss before we can apply those changes in main. I would like to discuss the following topics before merging:

  1. Autoencoder. In this comment, you suggested ravel pytree. But if I use ravel_pytree, the output is not the same, probably because of dealing with an 1D-array. So my approach was tree_flatten, concatenate and after the encode and decode step, we get the pytree back by using tree_unflatten. But here is my question: I used a very hacky solution to revert the concatenate function here. To easily use the split function we need split_vals to get the leaves back. I get those values here, which is also not great, so maybe we can discuss that.

  2. The overall structure of the project. I implemented a first draft. I renamed some files in the "tests" folder and created some new files. In main_test there is a new function called train_model. It is a universal function to train both, the real data and the synth data. In synth_data_test and real_data_test are pytests, so we can keep track of opt_tools and ros_datahandler. Everything related to the model can be found in the directory "model". In the directory "dataclass_creator" there is a helper function for creating/loading real and synth data. Also two utils directories. One in "dataclass_creator" and one in "encoder_dynamics".

  3. The Dataloader. Since I joined the project we used the Dataloader from pytorch to load the batches into the autoencoder. We created a collate function to fit the dataclass structure. IMO we can directly iter through our data if we adjust the shape of our attributes. That is what I did in create_loader. I changed the shape of each attribute into (num_batches, batch_size, window_size, num_features). I tested it and it works fine for the synth data (also faster because we don't need the collate function in each train step). I don't know if there are any benefits of the pytorch Dataloader that I haven't considered.

I will continue working and improving the structure, but maybe we can merge those changes into main, if everything is fine.

Merge request reports