"In general, reinforcement learning algorithms are based on the interaction of an agent with its environment. The agents observes a state of the environment and takes an action based on its policy. For each action the agent receives a feedback in form of a reward from the environment. The agents goal is to optimize its policy in order to gain maximum long-term reward.\n",
"The MuZero alogrithm belongs into the category of model-based reinforcement learning algorithms, and it combines a Monte Carlo Tree Search (MCTS) with a learned model. One great advantage when using a MuZero model, is that there are no direct constraints or requirements for the hidden-state to capure all the information of the environment. This makes the algorithm especially suitable to scenarios in which the environments dynamics are not fully known (such as for QML problems).\n",
"The conceputual layout graphic shows the basic workflow of the automatic feature map generation framework. \n",
"The current feature map configuration is observed by the agent, and then transformed into a hidden state representation. This hidden state representation is processed by a MCTS (gaining hidden rewards, value-, and policy functions). This is done by training classical neural networks, and the hidden state is updated iteratively. After the MCTS is propagated to a predefined depth a final policy is created and a final action (in form of the next gate of the quantum circuit) is sampled from the policy. \n",
...
...
%% Cell type:markdown id: tags:
# Automatic Quantum Feature Map Generation
***This demonstrator is part of the project [SEQUOIA End-to-End](https://websites.fraunhofer.de/sequoia/) of the [Competence Center Quantum Computing Baden-Württemberg](https://www.iaf.fraunhofer.de/en/networkers/KQC.html).***
Author: Frederic Rapp, Fraunhofer IPA, frederic.rapp@ipa.fraunhofer.de and David Kreplin, Fraunhofer IPA david.kreplin@ipa.fraunhofer.de
In this Notebook, we deomnstrate the ability to create problem-tailored quantum feature maps with a reinforcement learning algorithm.
When training Machine Learning (ML) models, it is already known that the model architecture has to be designed specifically for the problem at hand. Random models in general do not generalize well on unseen data.
When using a Quantum Machine Learning (QML) model, the essential building block of the model will be the parameterized quantum circuit (PQC) / quantum feature map. In the contemporary literature and research most feature map architectures depend on ad-hoc approaches. In order to gain a better performance we use a reinforcement learning algorithm to create a problem specific circuit dependent on the given data problem.
## Method
We use a powerful reinforcement learning method for the automatic generation of quantum feature maps, the MuZero algorithm [1].
In general, reinforcement learning algorithms are based on the interaction of an agent with its environment. The agents observes a state of the environment and takes an action based on its policy. For each action the agent receives a feedback in form of a reward from the environment. The agents goal is to optimize its policy in order to gain maximum long-term reward.
The MuZero alogrithm belongs into the category of model-based reinforcement learning algorithms, and it combines a Monte Carlo Tree Search (MCTS) with a learned model. One great advantage when using a MuZero model, is that there are no direct constraints or requirements for the hidden-state to capure all the information of the environment. This makes the algorithm especially suitable to scenarios in which the environments dynamics are not fully known (such as for QML problems).
The conceputual layout graphic shows the basic workflow of the automatic feature map generation framework.
The current feature map configuration is observed by the agent, and then transformed into a hidden state representation. This hidden state representation is processed by a MCTS (gaining hidden rewards, value-, and policy functions). This is done by training classical neural networks, and the hidden state is updated iteratively. After the MCTS is propagated to a predefined depth a final policy is created and a final action (in form of the next gate of the quantum circuit) is sampled from the policy.
Each feature map configuration is plugged into a Projected Quantum Kernel (PQK) [2], and the kernel is used in a QML model. The performance of the QML model on the training data serves as reward for the reinforcement learning algorithm.
NOTE: Reinforcement learning in general can have quite high computational costs, and in general its best to run the MuZero algorithm on GPUs if available.
## Further information and contact
[Fraunhofer IPA Team Quantum Computing](https://www.ipa.fraunhofer.de/quantum)
## General Imports
%% Cell type:code id: tags:
``` python
importnumpyasnp
# Get the MuZero and the Automatic Feature Map generator environment
In this demonstrator, we perform the automatic quantum feature map generation on the well known California Housing dataset.
The dataset contains a regression problem, the prediction of the respective house value.
The dataset contains 8 features consisting of information like, the age of the house, the average number of rooms, geographical position, etc.
In general the dataset consists of 20,640 samples. Due to a restriction in computational resources for QML, we use a subsample 1000 data points.
We split the dataset into 700 training points and 300 testing points. Additionaly due to the nature of QML models, we scale the all the data within a range of [-1,1].
In the following, we have to define our environment.
We need to specify the number of qubits we want to use (which is in general the number of features or n*(numer of features)).
Addtitonally, we need to define the maximum depth for the feature map, this parameter also serves as maximum depth of the MCTS.
As QML model we use a Quantum Support Vector Regression (QSVR) algorithm.
The environment, as well as the configuration files allow for a variety of changes in the hyperparameters of the algorithm.
Here we use a pre-trained model, feel free to add additional training steps by editing the num_steps parameter (but be aware that a high number of training steps comes with high additional computational cost).
%% Cell type:code id: tags:
``` python
num_steps=0# Set to 0 to only test the model,
# larger values will continue the training of the model
2024-03-27 15:53:24,139 INFO worker.py:1724 -- Started a local Ray instance.
Using checkpoint from chousing_cvscore_lac_new_scaling\results\results\model.checkpoint
Initializing replay buffer with chousing_cvscore_lac_new_scaling\results\results\replay_buffer.pkl
%% Cell type:markdown id: tags:
After loading the model can be trained further.
%% Cell type:code id: tags:
``` python
# Train the model if num_steps > 0
muzero.train()
```
%% Output
Training...
Run tensorboard --logdir ./results and go to http://localhost:6006/ to see in real time the training performance.
Last test reward: 143.00. Training step: 750000/750000. Played games: 18. Loss: 81.32
Shutting down workers...
Persisting replay buffer games to disk at C:\Users\DKR\Documents\Git Repositories\automatic-feature-map-generation\chousing_cvscore_lac_new_scaling\results\results\replay_buffer.pkl
[36m(ReplayBuffer pid=29100)[0m Replay buffer initialized with 165 samples (18 games).
[36m(ReplayBuffer pid=29100)[0m
[36m(Trainer pid=31184)[0m You are not training on GPU.
[36m(Trainer pid=31184)[0m Loading optimizer...
%% Cell type:markdown id: tags:
When the trained model converges (this can be seen when analyzing the seperate tensorboard ouputs) one can start calling the test method instead of the training routine, the MuZero networks weights are kept fixed and won't be trained any further. This allows to reduce the computational cost for the upcoming generations of generated feature maps.
The num_tests parameter controlls the amount of test runs. In general the more runs the better (since you will obtain even more feature map configurations to compare), since compuational resources for this notebook will be restricted we suggest to keep the number of test runs low.
The generation of the feature maps is visualized in the following animation, to allow the user to get some intuition about the generating process.
%% Cell type:code id: tags:
``` python
# Run test and create an animation of the created quantum feature maps
test=muzero.test(render=True,num_tests=3)
```
%% Output
Testing 1/3
Testing 2/3
Testing 3/3
%% Cell type:code id: tags:
``` python
get_animation()
```
%% Output
<IPython.core.display.HTML object>
%% Cell type:markdown id: tags:
Now we can load the result of the MuZero search algorithm (training & testing) to get all the generated feature maps which lead to a reward.
We will then extract the feature map which lead to the best solution (in this case to the best cross-validation score on the training data).
%% Cell type:code id: tags:
``` python
# Load results from training and testing and obtain the best found quantum feature map
In the final step, we use the best created feature map to perform a final QSVR on the test data!
We compare the result to a result of a QSVR based on a popular ad-hoc feature map, and addtionaly to a classical SVR.
For all models we perform a hyperparameter optimization using a grid search approach.
We use our in-house software package sQUlearn [3] for the QML methods. It allows for easy quantum computing backend inclusion with its Executor class, and the LayeredEncodingCircuit class comes in quite handy when building custom circuits architectures / quantum feature maps.
The framework also allows for easy high level implementation of QML models, rangeing from quantum kernel methods to quantum neural networks.
NOTE: The inference of the QML method takes up to 10 minutes for each model, due to the fairly high amout of data points for a quantum kernel method.
axs[1].set_title("QSVR based on ad-hoc feature map")
axs[2].set_title("QSVR based on generated feature map")
```
%% Output
best parameters for SVR with simulated Hubregtsen kernel: {'C': 0.01, 'epsilon': 1e-06}
best parameters for SVR with ad-hoc kernel: {'C': 10, 'epsilon': 0.0001}
best parameters for classical SVR: {'C': 0.1, 'epsilon': 1e-06}
Text(0.5, 1.0, 'QSVR based on generated feature map')
%% Cell type:markdown id: tags:
## Discussion
In this notebook, we provided a demonstration of the automatic quantum feature map generation framework. We use the state-of-the-art MuZero search algorithm which proves to be very succesful in building problem-tailored feature map designs.
The result above shows that the QML model based on a generated feature map leads to superior performance in comparance to a QML model based on an ad-hoc approach from contemporary literature [4].
%% Cell type:markdown id: tags:
## References
[1] Schrittwieser, J., Antonoglou, I., Hubert, T. et al. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature 588, 604–609 (2020). https://doi.org/10.1038/s41586-020-03051-4
[2] Huang, HY., Broughton, M., Mohseni, M. et al. Power of data in quantum machine learning. Nat Commun 12, 2631 (2021). https://doi.org/10.1038/s41467-021-22539-9
[3] Kreplin, D.A., Willmann, M., Schnabel, J., Rapp, F., Roth, M. sQUlearn – A Python Library for Quantum Machine Learning (2023). https://doi.org/10.48550/arXiv.2311.08990
[4] Thomas Hubregtsen, David Wierichs, Elies Gil-Fuster, Peter-Jan H. S. Derks, Paul K. Faehrmann, and Johannes Jakob Meyer
Phys. Rev. A 106, 042431 – Published 20 October 2022
