{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Otello AI\n",
    "\n",
    "The game reversi is a very good game to apply deep learning methods to.\n",
    "\n",
    "Othello, also known as reversi, is a board game first published in 1883 by either Lewis Waterman or John W. Mollet in England (each one was denouncing the other as fraud).\n",
    "It is a strict turn-based zero-sum game with a clear Markov chain and no hidden states, unlike card games with an unknown distribution of cards or unknown player allegiance.\n",
    "The game is played with one set of stones with two colors, which is much easier to abstract than chess with its 6 unique pieces.\n",
    "The game board is symmetrical and allows for playing with rotating the state around an axis or flipping/mirroring the board, which can allow for a breaking of sequences or interesting ANN architectures, quadruple the data generation by simulation, or interesting test cases where symmetry in turns should be observable if the AI reaches an \"objective\" policy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The game rules\n",
    "\n",
    "Othello is a turn-based, two-player board game played on an 8 x 8 board, with a similar geometry to a chess game. The game pieces are black on one side and white on the other.\n",
    "\n",
    "![Othello game board example](reversi_example.png)\n",
    "\n",
    "The players take turns placing their stones on the board, and the objective is to surround the opponent's stones with your own stones. A player can only place a stone when it surrounds at least one of the opponent's stones with their own stones, either horizontally, vertically, or diagonally. When a player places a stone, all the surrounded stones will flip to become the player's color. If a player cannot make a move, they are skipped. The game ends when both players cannot make any more moves. The player with the most stones on the board wins, and any unclaimed fields go to the player with the most stones of their color on the board. The game starts with four stones placed in the center of the board, with each player getting two, which are placed diagonally opposite to each other.\n",
    "\n",
    "\n",
    "<img alt=\"Startaufstellung.png\" src=\"Startaufstellung.png\"/>\n",
    "\n",
    "## Some common Othello strategies\n",
    "\n",
    "The placement of stones on the board is always a careful balance of attack and defense. Occupying large homogeneous stretches on the board can make it easier for the opponent to attack. The board's corners provide safety, from which occupied territory is impossible to lose, but they are difficult to obtain. The enemy must be forced to allow reaching the corners or calculate the cost of giving a stable base to the opponent. Some Othello computer strategies implement greedy algorithms based on a modified score for each field. Different values serve as score modifiers for a traditional greedy algorithm. When a player's stone captures a field, the score reached is multiplied by the modifier. The total score is the score reached by the player minus the score of the opponent. The scores change during the game and converge towards one, which gives some indications of what to expect from an Othello AI.\n",
    "\n",
    "<img alt=\"ComputerPossitionScore\" src=\"computer-score.png\"/>\n",
    "\n",
    "\n",
    "## Initial design decisions\n",
    "\n",
    "At the beginning of this project, I made some design decisions. The first one was that I did not want to use a gym library because it limits the data formats accessible. I chose to implement the whole game as an entry in a stack of NumPy arrays to be able to accommodate interfacing with a neural network easier and to use SciPy pattern recognition tools to implement some game mechanics for a fast simulation cycle. In the array format, stones from the player are marked as 1, and stones by the enemy are marked as -1. I chose to ignore player colors as far as I could; instead, a player perspective was used, which allowed changing the perspective with a flipping of the sign (multiplying with -1). The array format should also allow for data multiplication or the breaking of strict sequences by flipping the game along one of the four axes (horizontal, vertical, transpose along both diagonals).\n",
    "\n",
    "I wanted to implement different agents as classes that act on those game stacks. Since computation time is critical, all computational results are saved. The analysis of those is then repeated in real-time. If a recalculation of such a section is required, the save file can be deleted, and the code should be executed again.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:12.506631Z",
     "start_time": "2023-03-30T23:51:09.663807Z"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext blackcellmagic\n",
    "%load_ext line_profiler\n",
    "%load_ext memory_profiler"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports and dependencies\n",
    "\n",
    "The following direct dependencies where used for this project:\n",
    "```toml\n",
    "jupyter = \"^1.0.0\"\n",
    "matplotlib = \"^3.6.3\"\n",
    "numpy = \"^1.24.1\"\n",
    "pytest = \"^7.2.1\"\n",
    "python = \"3.10.*\"\n",
    "scipy = \"^1.10.0\"\n",
    "tqdm = \"^4.64.1\"\n",
    "jupyterlab = \"^3.6.1\"\n",
    "torchvision = \"^0.14.1\"\n",
    "torchaudio = \"^0.13.1\"\n",
    "```\n",
    "* `Jupyter` and `jupyterlab` on pycharm was used as an IDE / Ipython was used to implement this code.\n",
    "* `matplotlib` was used for visualisation and statistics.\n",
    "* `numpy` was used for array support and mathematical functions\n",
    "* `tqdm` was used for progress bars\n",
    "* `scipy` contains fast pattern recognition tools for images. It was used to make an initial estimation about where possible turns should be.\n",
    "* `torch` supplied the ANN functionalities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:12.569337Z",
     "start_time": "2023-03-30T23:51:09.705213Z"
    }
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "import abc\n",
    "import itertools\n",
    "import os.path\n",
    "from abc import ABC\n",
    "from enum import Enum\n",
    "from typing import Final\n",
    "from IPython.display import clear_output, display\n",
    "from pathlib import Path\n",
    "import glob\n",
    "import copy\n",
    "from functools import lru_cache, wraps\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.nn import functional\n",
    "from ipywidgets import interact\n",
    "from scipy.ndimage import binary_dilation\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constants\n",
    "\n",
    "Some general constants needed to be defined. Such as board game size and Player and Enemy representations. Also, directional offsets and the initial placement of blocks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:12.615852Z",
     "start_time": "2023-03-30T23:51:09.735610Z"
    }
   },
   "outputs": [],
   "source": [
    "BOARD_SIZE: Final[int] = 8  # defines the board side length as 8\n",
    "PLAYER: Final[int] = 1  # defines the number symbolising the player as 1\n",
    "ENEMY: Final[int] = -1  # defines the number symbolising the enemy as -1\n",
    "EXAMPLE_STACK_SIZE: Final[int] = 1000  # defines the game stack size for examples\n",
    "IMPOSSIBLE: Final[np.ndarray] = np.array([-1, -1], dtype=int)\n",
    "IMPOSSIBLE.setflags(write=False)\n",
    "SIMULATE_TURNS: Final[int] = 70\n",
    "VERIFY_POLICY: Final[bool] = False\n",
    "TRAINING_RESULT_PATH: Final[Path] = Path(\"training_data\")\n",
    "if not os.path.exists(TRAINING_RESULT_PATH):\n",
    "    os.mkdir(TRAINING_RESULT_PATH)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The directions array contains all the numerical offsets needed to move along one of the 8 directions in a two-dimensional grid. This will allow an iteration over the game board.\n",
    "\n",
    "![8-directions.png](8-directions.png \"Offset in 8 directions\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:12.653589Z",
     "start_time": "2023-03-30T23:51:09.784263Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1, -1],\n",
       "       [-1,  0],\n",
       "       [-1,  1],\n",
       "       [ 0, -1],\n",
       "       [ 0,  1],\n",
       "       [ 1, -1],\n",
       "       [ 1,  0],\n",
       "       [ 1,  1]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DIRECTIONS: Final[np.ndarray] = np.array(\n",
    "    [[i, j] for i in range(-1, 2) for j in range(-1, 2) if j != 0 or i != 0],\n",
    "    dtype=int,\n",
    ")\n",
    "DIRECTIONS.setflags(write=False)\n",
    "DIRECTIONS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another constant needed is the initial start square at the center of the board."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:12.680563Z",
     "start_time": "2023-03-30T23:51:09.819690Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-1,  1],\n",
       "       [ 1, -1]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "START_SQUARE: Final[np.ndarray] = np.array(\n",
    "    [[ENEMY, PLAYER], [PLAYER, ENEMY]], dtype=int\n",
    ")\n",
    "START_SQUARE.setflags(write=False)\n",
    "START_SQUARE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating new boards\n",
    "\n",
    "The first function implemented and tested is a function to generate the starting environment as a stack of games.\n",
    "As described above I simply placed a 2 by 2 square in the center of an empty stack of boards."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:12.745419Z",
     "start_time": "2023-03-30T23:51:09.855798Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0,  0,  0,  0,  0,  0,  0,  0],\n",
       "       [ 0,  0,  0,  0,  0,  0,  0,  0],\n",
       "       [ 0,  0,  0,  0,  0,  0,  0,  0],\n",
       "       [ 0,  0,  0, -1,  1,  0,  0,  0],\n",
       "       [ 0,  0,  0,  1, -1,  0,  0,  0],\n",
       "       [ 0,  0,  0,  0,  0,  0,  0,  0],\n",
       "       [ 0,  0,  0,  0,  0,  0,  0,  0],\n",
       "       [ 0,  0,  0,  0,  0,  0,  0,  0]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def get_new_games(number_of_games: int) -> np.ndarray:\n",
    "    \"\"\"Generates a stack of initialised game boards.\n",
    "\n",
    "    Args:\n",
    "        number_of_games: The size of the board stack.\n",
    "\n",
    "    Returns: The generates stack of games as a stack n x 8 x 8.\n",
    "\n",
    "    \"\"\"\n",
    "    empty = np.zeros([number_of_games, BOARD_SIZE, BOARD_SIZE], dtype=int)\n",
    "    empty[:, 3:5, 3:5] = START_SQUARE\n",
    "    return empty\n",
    "\n",
    "\n",
    "get_new_games(1)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:12.776309Z",
     "start_time": "2023-03-30T23:51:09.873794Z"
    }
   },
   "outputs": [],
   "source": [
    "test_number_of_games = 3\n",
    "assert get_new_games(test_number_of_games).shape == (\n",
    "    test_number_of_games,\n",
    "    BOARD_SIZE,\n",
    "    BOARD_SIZE,\n",
    ")\n",
    "np.testing.assert_equal(\n",
    "    get_new_games(test_number_of_games).sum(axis=1),\n",
    "    np.zeros(\n",
    "        [\n",
    "            test_number_of_games,\n",
    "            8,\n",
    "        ]\n",
    "    ),\n",
    ")\n",
    "np.testing.assert_equal(\n",
    "    get_new_games(test_number_of_games).sum(axis=2),\n",
    "    np.zeros(\n",
    "        [\n",
    "            test_number_of_games,\n",
    "            8,\n",
    "        ]\n",
    "    ),\n",
    ")\n",
    "assert np.all(get_new_games(test_number_of_games)[:, 3:4, 3:4] != 0)\n",
    "del test_number_of_games"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualisation tools\n",
    "\n",
    "In this section a visualisation help was implemented for debugging of the game and a proper display of the results.\n",
    "For this visualisation ChatGPT was used as a prompted code generator that was later reviewed and refactored by hand to integrate seamlessly into the project as a whole.\n",
    "White stones represent the player, black stones the enemy. A single plot can be used as a subplot when the `ax` argument is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:12.915811Z",
     "start_time": "2023-03-30T23:51:09.900138Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 12952 characters]\n",
      "text/plain": [
       "<Figure size 300x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_othello_board(\n",
    "    board: np.ndarray | torch.Tensor,\n",
    "    action: np.ndarray | None = None,\n",
    "    score: float | None = None,\n",
    "    ax=None,\n",
    ") -> None:\n",
    "    \"\"\"Plots a single otello board.\n",
    "\n",
    "    If a matplot axis object is given the board will be plotted into that axis. If not an axis object will be generated.\n",
    "    The image generated will be shown directly.\n",
    "\n",
    "    Args:\n",
    "        board: The bord that should be plotted. Only a single games is allowed. A numpy array of the form 8x8 is expected.\n",
    "        action: The action taken on each board.\n",
    "        score: The score reached with the turn.\n",
    "        ax: If needed a matplotlib axis object can be defined that is used to place the board as a sublot into a bigger context.\n",
    "    \"\"\"\n",
    "    # convert a tensor into an array\n",
    "    if isinstance(board, torch.Tensor):\n",
    "        board = board.cpu().detach().numpy()\n",
    "\n",
    "    # ensure the shape of the array fits\n",
    "    assert board.shape == (8, 8)\n",
    "    plot_all = False\n",
    "\n",
    "    # create a figure if no axis is given\n",
    "    if ax is None:\n",
    "        fig_size = 3\n",
    "        plot_all = True\n",
    "        fig, ax = plt.subplots(figsize=(fig_size, fig_size))\n",
    "\n",
    "    # set the background color\n",
    "    ax.set_facecolor(\"#0f6b28\")\n",
    "\n",
    "    # plot the actions\n",
    "    if action is not None:\n",
    "        ax.scatter(action[0], action[1], s=350 if plot_all else 200, c=\"red\")\n",
    "\n",
    "    # plot black and white stones\n",
    "    for x_pos, y_pos in itertools.product(range(BOARD_SIZE), range(BOARD_SIZE)):\n",
    "        if board[x_pos, y_pos] == ENEMY:\n",
    "            color = \"white\"\n",
    "        elif board[x_pos, y_pos] == PLAYER:\n",
    "            color = \"black\"\n",
    "        else:\n",
    "            continue\n",
    "        ax.scatter(x_pos, y_pos, s=260 if plot_all else 140, c=color)\n",
    "\n",
    "    # plot the lines separating the fields\n",
    "    for x_pos in range(-1, 8):\n",
    "        ax.axhline(x_pos + 0.5, color=\"black\", lw=2)\n",
    "        ax.axvline(x_pos + 0.5, color=\"black\", lw=2)\n",
    "\n",
    "    # define the size of the plot\n",
    "    ax.set_xlim(-0.5, 7.5)\n",
    "    ax.set_ylim(7.5, -0.5)\n",
    "\n",
    "    # set the axis labels\n",
    "    ax.set_xticks(np.arange(8))\n",
    "    ax.set_xticklabels(list(\"ABCDEFGH\"))\n",
    "    ax.set_yticks(np.arange(8))\n",
    "    ax.set_yticklabels(list(\"12345678\"))\n",
    "\n",
    "    # overrides the x_label text if a score should be plotted\n",
    "    if score is None:\n",
    "        ax.set_xlabel(\n",
    "            f\"B{np.sum(board == PLAYER)} / {np.sum(board == 0)} / W{np.sum(board == ENEMY)}\"\n",
    "        )\n",
    "    else:\n",
    "        ax.set_xlabel(f\"Score: {score}\")\n",
    "    if plot_all:\n",
    "        plt.tight_layout()\n",
    "\n",
    "\n",
    "plot_othello_board(get_new_games(1)[0], action=np.array([3, 3]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:12.916815Z",
     "start_time": "2023-03-30T23:51:10.199792Z"
    }
   },
   "outputs": [],
   "source": [
    "PLOTS_PER_ROW = 4\n",
    "\n",
    "\n",
    "def plot_othello_boards(\n",
    "    boards: np.ndarray,\n",
    "    actions: np.ndarray | None = None,\n",
    "    scores: np.ndarray | None = None,\n",
    ") -> None:\n",
    "    \"\"\"Plots multiple boards into subplots.\n",
    "\n",
    "    The plots are shown directly.\n",
    "\n",
    "    Args:\n",
    "        boards: Plots the boards given into subplots. The maximum number of boards accepted is 70.\n",
    "        actions: A list of actions taken on each of the boards.\n",
    "        scores: A list of scores reached at each board.\n",
    "    \"\"\"\n",
    "    # checking if the array input shapes do fit\n",
    "    assert len(boards.shape) == 3\n",
    "    assert boards.shape[1:] == (BOARD_SIZE, BOARD_SIZE)\n",
    "    assert boards.shape[0] < 70\n",
    "\n",
    "    if actions is not None:\n",
    "        assert len(actions.shape) == 2\n",
    "        assert actions.shape[1] == 2\n",
    "        assert boards.shape[0] == actions.shape[0]\n",
    "\n",
    "    if scores is not None:\n",
    "        assert len(scores.shape) == 1\n",
    "        assert boards.shape[0] == scores.shape[0]\n",
    "\n",
    "    # plots the boards\n",
    "    rows = int(np.ceil(boards.shape[0] / PLOTS_PER_ROW))\n",
    "    fig, axs = plt.subplots(rows, PLOTS_PER_ROW, figsize=(12, 3 * rows))\n",
    "    for game_index, ax in enumerate(axs.flatten()):\n",
    "        if game_index >= boards.shape[0]:\n",
    "            fig.delaxes(ax)\n",
    "        else:\n",
    "            action = actions[game_index] if actions is not None else None\n",
    "            score = scores[game_index] if scores is not None else None\n",
    "            plot_othello_board(boards[game_index], action=action, score=score, ax=ax)\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:12.916815Z",
     "start_time": "2023-03-30T23:51:10.215799Z"
    }
   },
   "outputs": [],
   "source": [
    "def drop_duplicate_boards(\n",
    "    boards: np.ndarray,\n",
    "    actions: np.ndarray | None,\n",
    ") -> tuple[np.ndarray, np.ndarray | None]:\n",
    "    \"\"\"Takes a sequence of boards and drops all boards that are unchanged.\n",
    "\n",
    "    Args:\n",
    "        boards: A list of boards to be reduced.\n",
    "        actions: A list of actions to be reduced alongside the bords.\n",
    "\n",
    "    Returns:\n",
    "        A sequence of boards where boards that where equal are dropped.\n",
    "    \"\"\"\n",
    "    non_duplicates = ~np.all(boards == np.roll(boards, axis=0, shift=1), axis=(1, 2))\n",
    "    return (\n",
    "        boards[non_duplicates],\n",
    "        np.roll(actions, axis=0, shift=1)[non_duplicates]\n",
    "        if actions is not None\n",
    "        else None,\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hash Otello Boards\n",
    "\n",
    "A challenge for training any reinforcement learning algorithm is how to properly calibrate the exploration rate.\n",
    "To make huge numbers of boards comparable it is easier to work with hashes than with the actual boards. For that purpose a functionality to hash a board and a stack of boards was added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:12.992902Z",
     "start_time": "2023-03-30T23:51:10.232792Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def hash_board(board: np.ndarray) -> int:\n",
    "    assert board.shape == (8, 8) or board.shape == (64,)\n",
    "    return hash(tuple(board.reshape(-1)))\n",
    "\n",
    "\n",
    "def count_unique_boards(boards: np.ndarray) -> int:\n",
    "    return np.unique(\n",
    "        np.apply_along_axis(hash_board, axis=1, arr=boards.reshape(-1, 64))\n",
    "    ).size\n",
    "\n",
    "\n",
    "a = count_unique_boards(np.random.randint(-1, 2, size=(10_000, 8, 8)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find possible actions to take\n",
    "\n",
    "The frist step in the implementation of an AI like this is to get an overview over the possible actions that can be taken in a situation.\n",
    "Here was the design choice taken to first find fields that are empty and have at least one neighbouring enemy stone.\n",
    "This was implemented with element wise check for a stone and a binary dilation marking all fields neighboring an enemy stone.\n",
    "For that the `SURROUNDING` mask was used. Both aries are then element wise combined using and.\n",
    "The resulting array contains all filed where a turn could potentially be made. Those are then check in detail.\n",
    "The previous element wise operations on the numpy array increase the spead for this operation dramatically.\n",
    "\n",
    "The check for a possible turn is done in detail by following each direction step by step as long as there are enemy stones in that direction.\n",
    "If the board end is reached or en empty filed before reaching a field occupied by the player that direction does not surround enemy stones.\n",
    "If one direction surrounds enemy stone a turn is possible.\n",
    "This detailed step is implemented as a recursion and need to go at leas one step to return True."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:13.002888Z",
     "start_time": "2023-03-30T23:51:10.362177Z"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[1, 1, 1],\n",
       "        [1, 0, 1],\n",
       "        [1, 1, 1]]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "SURROUNDING: Final = np.array(\n",
    "    [[[1, 1, 1], [1, 0, 1], [1, 1, 1]]]\n",
    ")  # defines the binary dilation mask to check if a field is next to an enemy stones\n",
    "SURROUNDING"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To optimize the computation of this game, the `lru_cache` decorator was utilized. LRU cache stores the hash of the arguments and returns the previously calculated result of a computationally heavy operation. However, since Numpy arrays are mutable and unhashable, a code snippet was modified to include conversion to tuples, caching layer, and reconversion to Numpy arrays. This allows for the caching to be implemented. As a result, the calculation time of possible actions to take was reduced to only 30% of the time it takes without the lru_cache decorator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:13.003895Z",
     "start_time": "2023-03-30T23:51:10.380173Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Source https://gist.github.com/Susensio/61f4fee01150caaac1e10fc5f005eb75\n",
    "def np_cache(*lru_args, array_argument_elements: tuple[int, ...] = (0,), **lru_kwargs):\n",
    "    \"\"\"\n",
    "    LRU cache implementation for functions whose parameter at ``array_argument_index`` is a numpy array of dimensions <= 2\n",
    "\n",
    "    Example:\n",
    "    >>> from sem_env.utils.cache import np_cache\n",
    "    >>> array = np.array([[1, 2, 3], [4, 5, 6]])\n",
    "    >>> @np_cache(maxsize=256)\n",
    "    ... def multiply(array, factor):\n",
    "    ...     return factor * array\n",
    "    >>> multiply(array, 2)\n",
    "    >>> multiply(array, 2)\n",
    "    >>> multiply.cache_info()\n",
    "    CacheInfo(hits=1, misses=1, maxsize=256, currsize=1)\n",
    "    \"\"\"\n",
    "\n",
    "    def decorator(function):\n",
    "        @wraps(function)\n",
    "        def wrapper(*args, **kwargs):\n",
    "            args = list(args)\n",
    "            for array_argument_index in array_argument_elements:\n",
    "                np_array = args[array_argument_index]\n",
    "                if len(np_array.shape) > 2:\n",
    "                    raise RuntimeError(\n",
    "                        f\"np_cache is currently only supported for arrays of dim. less than 3 but got shape: {np_array.shape}\"\n",
    "                    )\n",
    "                hashable_array = tuple(np_array.reshape(-1))\n",
    "\n",
    "                args[array_argument_index] = hashable_array\n",
    "            return cached_wrapper(*args, **kwargs)\n",
    "\n",
    "        @lru_cache(*lru_args, **lru_kwargs)\n",
    "        def cached_wrapper(*args, **kwargs):\n",
    "            args = list(args)\n",
    "            for array_argument_index in array_argument_elements:\n",
    "                hashable_array = args[array_argument_index]\n",
    "                array = np.array(hashable_array).reshape(8, 8)\n",
    "                args[array_argument_index] = array\n",
    "            return function(*args, **kwargs)\n",
    "\n",
    "        # copy lru_cache attributes over too\n",
    "        wrapper.cache_info = cached_wrapper.cache_info\n",
    "        wrapper.cache_clear = cached_wrapper.cache_clear\n",
    "        return wrapper\n",
    "\n",
    "    return decorator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:30.997965Z",
     "start_time": "2023-03-30T23:51:10.405144Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "194 µs ± 9.18 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[[False, False, False, False, False, False, False, False],\n",
       "        [False, False, False, False, False, False, False, False],\n",
       "        [False, False, False,  True, False, False, False, False],\n",
       "        [False, False,  True, False, False, False, False, False],\n",
       "        [False, False, False, False, False,  True, False, False],\n",
       "        [False, False, False, False,  True, False, False, False],\n",
       "        [False, False, False, False, False, False, False, False],\n",
       "        [False, False, False, False, False, False, False, False]]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def _recursive_steps(\n",
    "    board: np.ndarray,\n",
    "    rec_direction: np.ndarray,\n",
    "    rec_position: np.ndarray,\n",
    "    step_one: int = 0,\n",
    ") -> int:\n",
    "    \"\"\"Check if a player can place a stone on the board specified in the direction specified and direction specified.\n",
    "\n",
    "    Args:\n",
    "        board: The board that should be checked for a playable action.\n",
    "        rec_direction: The direction that should be checked.\n",
    "        rec_position: The position that should be checked.\n",
    "        step_one: Defines if the call of this function is the firs or not. Should be kept to the default value for proper functionality.\n",
    "\n",
    "    Returns:\n",
    "        True if a turn is possible for possition and direction on the board defined.\n",
    "    \"\"\"\n",
    "    rec_position = rec_position + rec_direction\n",
    "    if np.any((rec_position >= BOARD_SIZE) | (rec_position < 0)):\n",
    "        return 0\n",
    "    next_field = board[tuple(rec_position.tolist())]\n",
    "    if next_field == 0:\n",
    "        return 0\n",
    "    if next_field == -1:\n",
    "        return _recursive_steps(\n",
    "            board, rec_direction, rec_position, step_one=step_one + 1\n",
    "        )\n",
    "    if next_field == 1:\n",
    "        return step_one\n",
    "\n",
    "\n",
    "@np_cache(maxsize=2000, array_argument_elements=(0, 1))\n",
    "def _get_possible_turns_for_board(\n",
    "    board: np.ndarray, poss_turns: np.ndarray\n",
    ") -> np.ndarray:\n",
    "    \"\"\"Calculates where turns are possible.\n",
    "\n",
    "    Args:\n",
    "        board: The board that should be checked for a playable action.\n",
    "        poss_turns: An array of actions that could be possible. All true fields are empty and next to an enemy stone.\n",
    "    \"\"\"\n",
    "    for idx, idy in itertools.product(range(BOARD_SIZE), range(BOARD_SIZE)):\n",
    "        if poss_turns[idx, idy]:\n",
    "            position = idx, idy\n",
    "            poss_turns[idx, idy] = any(\n",
    "                _recursive_steps(board[:, :], direction, position) > 0\n",
    "                for direction in DIRECTIONS\n",
    "            )\n",
    "    return poss_turns\n",
    "\n",
    "\n",
    "def get_possible_turns(boards: np.ndarray, tqdm_on: bool = False) -> np.ndarray:\n",
    "    \"\"\"Analyses a stack of boards.\n",
    "\n",
    "    Args:\n",
    "        boards: A stack of boards to check.\n",
    "        tqdm_on: Uses tqdm to track the progress.\n",
    "\n",
    "    Returns:\n",
    "        A stack of game boards containing boolean values showing where turns are possible for the player.\n",
    "    \"\"\"\n",
    "    assert len(boards.shape) == 3, \"The number fo input dimensions does not fit.\"\n",
    "    assert boards.shape[1:] == (\n",
    "        BOARD_SIZE,\n",
    "        BOARD_SIZE,\n",
    "    ), \"The input dimensions do not fit.\"\n",
    "\n",
    "    poss_turns = boards == 0  # checks where fields are empty.\n",
    "    poss_turns &= binary_dilation(\n",
    "        boards == -1, SURROUNDING\n",
    "    )  # checks where fields are next to an enemy filed an empty\n",
    "    iterate_over = range(boards.shape[0])\n",
    "\n",
    "    if tqdm_on:\n",
    "        iterate_over = tqdm(iterate_over, total=np.prod(boards.shape))\n",
    "    for game in iterate_over:\n",
    "        poss_turns[game] = _get_possible_turns_for_board(boards[game], poss_turns[game])\n",
    "    return poss_turns\n",
    "\n",
    "\n",
    "# some simple testing to ensure the function works after simple changes\n",
    "# this testing is complete, its more of a smoke-test\n",
    "test_array = get_new_games(3)\n",
    "expected_result = np.zeros_like(test_array, dtype=bool)\n",
    "expected_result[:, 4, 5] = expected_result[:, 2, 3] = True\n",
    "expected_result[:, 5, 4] = expected_result[:, 3, 2] = True\n",
    "np.testing.assert_equal(get_possible_turns(test_array), expected_result)\n",
    "\n",
    "\n",
    "%timeit get_possible_turns(get_new_games(10))  # checks turn possibility evaluation time for 10 initial games\n",
    "# %timeit get_possible_turns(get_new_games(EXAMPLE_STACK_SIZE))  # check turn possibility evaluation time for EXAMPLE_STACK_SIZE initial games\n",
    "\n",
    "# shows a singe game\n",
    "get_possible_turns(get_new_games(3))[:1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Besides the ability to generate an array of possible turns there needs to be a functions that check if a given turn is possible.\n",
    "On is needed for the action space validation. The other is for validating a players turn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:31.089196Z",
     "start_time": "2023-03-30T23:51:31.003968Z"
    }
   },
   "outputs": [],
   "source": [
    "def move_possible(board: np.ndarray, move: np.ndarray) -> bool:\n",
    "    \"\"\"Checks if a turn is possible.\n",
    "\n",
    "    Checks if a turn is possible. If no turn is possible to input array [-1, -1] is expected.\n",
    "\n",
    "    Args:\n",
    "        board: A board where it should be checkt if a turn is possible.\n",
    "        move: The move that should be taken. Expected is the index of the filed where a stone should be placed [x, y]. If no placement is possible [-1, -1] is expected as an input.\n",
    "\n",
    "    Returns:\n",
    "        True if the move is possible\n",
    "    \"\"\"\n",
    "    if np.all(move == -1):\n",
    "        return not np.any(get_possible_turns(np.reshape(board, (1, 8, 8))))\n",
    "    return any(\n",
    "        _recursive_steps(board[:, :], direction, move) > 0 for direction in DIRECTIONS\n",
    "    )\n",
    "\n",
    "\n",
    "# Some testing for this function and the underlying recursive functions that are called.\n",
    "assert move_possible(get_new_games(1)[0], np.array([2, 3])) is True\n",
    "assert move_possible(get_new_games(1)[0], np.array([3, 2])) is True\n",
    "assert move_possible(get_new_games(1)[0], np.array([2, 2])) is False\n",
    "assert move_possible(np.zeros((8, 8)), np.array([3, 2])) is False\n",
    "assert move_possible(np.ones((8, 8)) * 1, np.array([-1, -1])) is True\n",
    "assert move_possible(np.ones((8, 8)) * -1, np.array([-1, -1])) is True\n",
    "assert move_possible(np.ones((8, 8)) * 0, np.array([-1, -1])) is True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:31.106199Z",
     "start_time": "2023-03-30T23:51:31.021960Z"
    }
   },
   "outputs": [],
   "source": [
    "def moves_possible(boards: np.ndarray, moves: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"Checks if a stack of moves can be executed on a stack of boards.\n",
    "\n",
    "    Args:\n",
    "        boards: A board where the next stone should be placed.\n",
    "        moves: A stack stones to be placed. Each move is formatted as an array in the form of [x, y] if no turn is possible the value [-1, -1] is expected.\n",
    "\n",
    "    Returns:\n",
    "        An array marking for each and every game and move in the stack if the move can be executed.\n",
    "    \"\"\"\n",
    "    arr_moves_possible = np.zeros(boards.shape[0], dtype=bool)\n",
    "    for game in range(boards.shape[0]):\n",
    "        if np.all(\n",
    "            moves[game] == -1\n",
    "        ):  # can be all or any. All should be faster since most times neither value will be -1.\n",
    "            arr_moves_possible[game] = not np.any(\n",
    "                get_possible_turns(np.reshape(boards[game], (1, 8, 8)))\n",
    "            )\n",
    "        else:\n",
    "            arr_moves_possible[game] = any(\n",
    "                _recursive_steps(boards[game, :, :], direction, moves[game]) > 0\n",
    "                for direction in DIRECTIONS\n",
    "            )\n",
    "    return arr_moves_possible\n",
    "\n",
    "\n",
    "np.testing.assert_array_equal(\n",
    "    moves_possible(np.ones((3, 8, 8)) * 1, np.array([[-1, -1]] * 3)),\n",
    "    np.array([True] * 3),\n",
    ")\n",
    "\n",
    "np.testing.assert_array_equal(\n",
    "    moves_possible(get_new_games(3), np.array([[2, 3], [3, 2], [3, 2]])),\n",
    "    np.array([True] * 3),\n",
    ")\n",
    "np.testing.assert_array_equal(\n",
    "    moves_possible(get_new_games(3), np.array([[2, 2], [1, 1], [0, 0]])),\n",
    "    np.array([False] * 3),\n",
    ")\n",
    "np.testing.assert_array_equal(\n",
    "    moves_possible(np.ones((3, 8, 8)) * -1, np.array([[-1, -1]] * 3)),\n",
    "    np.array([True] * 3),\n",
    ")\n",
    "np.testing.assert_array_equal(\n",
    "    moves_possible(np.zeros((3, 8, 8)), np.array([[-1, -1]] * 3)),\n",
    "    np.array([True] * 3),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execute a chosen action\n",
    "\n",
    "After an evaluation what turns are possible there needs to be a function that executes a turn.\n",
    "This next sections does that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:31.106199Z",
     "start_time": "2023-03-30T23:51:31.046196Z"
    }
   },
   "outputs": [],
   "source": [
    "class InvalidTurn(ValueError):\n",
    "    \"\"\"\n",
    "    This error is thrown if a given turn is not valid.\n",
    "    \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:42.365334Z",
     "start_time": "2023-03-30T23:51:31.074204Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "114 ms ± 3.18 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 12712 characters]\n",
      "text/plain": [
       "<Figure size 300x300 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def do_moves(boards: np.ndarray, moves: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"Executes a single move on a stack o Othello boards.\n",
    "\n",
    "    Args:\n",
    "        boards: A stack of Othello boards where the next stone should be placed.\n",
    "        moves: A stack of stone placement orders for the game. Formatted as coordinates in an array [x, y] of the place where the stone should be placed. Should contain [-1,-1] if no new placement is possible.\n",
    "\n",
    "    Returns:\n",
    "        The new state of the board.\n",
    "    \"\"\"\n",
    "\n",
    "    def _do_directional_move(\n",
    "        board: np.ndarray, rec_move: np.ndarray, rev_direction, step_one=True\n",
    "    ) -> bool:\n",
    "        \"\"\"Changes the color of enemy stones in one direction.\n",
    "\n",
    "        This function works recursive. The argument step_one should always be used in its default value.\n",
    "\n",
    "        Args:\n",
    "            board: A bord on which a stone was placed.\n",
    "            rec_move: The position on the board in x and y where this function is called from. Will be moved by recursive called.\n",
    "            rev_direction: The position where the stone was placed. Inside this recursion it will also be the last step that was checked.\n",
    "            step_one: Set to true if this is the first step in the recursion. False later on.\n",
    "\n",
    "        Returns:\n",
    "            True if a stone could be flipped.\n",
    "            All changes are made on the view of the numpy array and therefore not included in the return value.\n",
    "        \"\"\"\n",
    "        rec_position = rec_move + rev_direction\n",
    "        if np.any((rec_position >= 8) | (rec_position < 0)):\n",
    "            return False\n",
    "        next_field = board[tuple(rec_position.tolist())]\n",
    "        if next_field == 0:\n",
    "            return False\n",
    "        if next_field == 1:\n",
    "            return not step_one\n",
    "        if next_field == -1:\n",
    "            if _do_directional_move(board, rec_position, rev_direction, step_one=False):\n",
    "                board[tuple(rec_position.tolist())] = 1\n",
    "                return True\n",
    "            return False\n",
    "\n",
    "    def _do_move(_board: np.ndarray, move: np.ndarray) -> None:\n",
    "        \"\"\"Executes a turn on a board.\n",
    "\n",
    "        Args:\n",
    "            _board: The game board on wich to place a stone.\n",
    "            move: The coordinates of a stone that should be placed. Should be formatted as an array of the form [x, y]. The value [-1, -1] is expected if no turn is possible.\n",
    "\n",
    "        Returns:\n",
    "            All changes are made on the view of the numpy array.\n",
    "        \"\"\"\n",
    "        if np.all(move == -1):\n",
    "            if not move_possible(_board, move):\n",
    "                raise InvalidTurn(\"An action should be taken. A turn is possible.\")\n",
    "            return\n",
    "\n",
    "        # noinspection PyTypeChecker\n",
    "        if _board[tuple(move.tolist())] != 0:\n",
    "            raise InvalidTurn(\"This turn is not possible.\")\n",
    "\n",
    "        action = False\n",
    "        for direction in DIRECTIONS:\n",
    "            if _do_directional_move(_board, move, direction):\n",
    "                action = True\n",
    "        if not action:\n",
    "            raise InvalidTurn(\"This turn is not possible.\")\n",
    "\n",
    "        # noinspection PyTypeChecker\n",
    "        _board[tuple(move.tolist())] = 1\n",
    "\n",
    "    boards = boards.copy()\n",
    "    for game in range(boards.shape[0]):\n",
    "        _do_move(boards[game], moves[game])\n",
    "    return boards\n",
    "\n",
    "\n",
    "%timeit do_moves(get_new_games(EXAMPLE_STACK_SIZE), np.array([[2, 3]] * EXAMPLE_STACK_SIZE))[0]\n",
    "\n",
    "plot_othello_board(\n",
    "    do_moves(\n",
    "        get_new_games(EXAMPLE_STACK_SIZE), np.array([[2, 3]] * EXAMPLE_STACK_SIZE)\n",
    "    )[0],\n",
    "    action=np.array([2, 3]),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An abstract reversi game policy\n",
    "\n",
    "For an easy use of policies an abstract class containing the policy generation / requests an action in an inherited instance of this class.\n",
    "This class filters the policy to only propose valid actions. Inherited instance do not need to care about this. This super class also manges exploration and exploitation with the epsilon value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:42.422334Z",
     "start_time": "2023-03-30T23:51:42.374342Z"
    }
   },
   "outputs": [],
   "source": [
    "class GamePolicy(ABC):\n",
    "    \"\"\"\n",
    "    A game policy. Proposes where to place a stone next.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, epsilon: float):\n",
    "        \"\"\"\n",
    "\n",
    "        Args:\n",
    "            epsilon: the epsilon / greedy value. Should be between zero and one. Set the mixture of policy and exploration. One means only the policy is used. Zero means only random policies are used. All mixtures inbetween between are possible.\n",
    "        \"\"\"\n",
    "        if 0 > epsilon > 1:\n",
    "            raise ValueError(\"Epsilon should be between zero and one.\")\n",
    "        self.epsilon: float = epsilon\n",
    "\n",
    "    @property\n",
    "    @abc.abstractmethod\n",
    "    def policy_name(self) -> str:\n",
    "        \"\"\"The name of this policy\"\"\"\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    @abc.abstractmethod\n",
    "    def _internal_policy(self, boards: np.ndarray) -> np.ndarray:\n",
    "        \"\"\"The internal policy is an unfiltered policy. It should only be called from inside this function\n",
    "\n",
    "        Args:\n",
    "            boards: A board where a policy should be calculated for.\n",
    "\n",
    "        Returns:\n",
    "            The policy for this board. Should have the same size as the boards array.\n",
    "        \"\"\"\n",
    "        raise NotImplementedError()\n",
    "\n",
    "    def get_policy(self, boards: np.ndarray) -> np.ndarray:\n",
    "        \"\"\"Calculates the policy that should be followed.\n",
    "\n",
    "        Calculates the policy that should be followed.\n",
    "        This function does include the usage of epsilon to configure greediness and exploration.\n",
    "\n",
    "        Args:\n",
    "            boards: A set of boards that show the environment where the policy should be calculated for.\n",
    "\n",
    "        Returns:\n",
    "            A vector of indices. Should be formatted as an array of the form [x, y]. The value [-1, -1] is expected if no turn is possible.\n",
    "        \"\"\"\n",
    "        assert len(boards.shape) == 3\n",
    "        assert boards.shape[1:] == (BOARD_SIZE, BOARD_SIZE)\n",
    "\n",
    "        if self.epsilon <= 0:\n",
    "            policies = np.random.rand(*boards.shape)\n",
    "        else:\n",
    "            policies = self._internal_policy(boards)\n",
    "            if self.epsilon < 1:\n",
    "                random_choices = self.epsilon <= np.random.rand((boards.shape[0]))\n",
    "                policies[random_choices] = np.random.rand(np.sum(random_choices), 8, 8)\n",
    "\n",
    "        # todo possibly change this function to only validate the purpose turn and not all turns\n",
    "        possible_turns = get_possible_turns(boards)\n",
    "        policies[possible_turns == False] = -1.0\n",
    "        max_indices = [\n",
    "            np.unravel_index(policy.argmax(), policy.shape) for policy in policies\n",
    "        ]\n",
    "        policy_vector = np.array(max_indices, dtype=int)\n",
    "        no_turn_possible = np.all(policy_vector == 0, 1) & (policies[:, 0, 0] == -1.0)\n",
    "\n",
    "        policy_vector[no_turn_possible, :] = IMPOSSIBLE\n",
    "        return policy_vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some first policies\n",
    "\n",
    "To quantify the quality of a game AI there needs to be some benchmarks.\n",
    "The easiest benchmark is to play against a random player.\n",
    "The easiest player to use as a benchmark is the random player.\n",
    "For this and testing purpose the random policy was implemented."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:42.424333Z",
     "start_time": "2023-03-30T23:51:42.400400Z"
    }
   },
   "outputs": [],
   "source": [
    "class RandomPolicy(GamePolicy):\n",
    "    \"\"\"\n",
    "    A policy playing a random turn by setting epsilon to 0.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, epsilon: float = 0):\n",
    "        _ = epsilon\n",
    "        super().__init__(epsilon=0)\n",
    "\n",
    "    @property\n",
    "    def policy_name(self) -> str:\n",
    "        return \"random\"\n",
    "\n",
    "    def _internal_policy(self, boards: np.ndarray) -> np.ndarray:\n",
    "        return np.random.rand(*boards.shape)\n",
    "\n",
    "\n",
    "rnd_policy = RandomPolicy(1)\n",
    "assert rnd_policy.policy_name == \"random\"\n",
    "assert rnd_policy.epsilon == 0\n",
    "\n",
    "rnd_policy_result = rnd_policy.get_policy(get_new_games(10))\n",
    "assert np.any((5 >= rnd_policy_result) & (rnd_policy_result >= 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An alternative benchmark policy is a greedy policy that takes always the maximum number of stones possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:42.453635Z",
     "start_time": "2023-03-30T23:51:42.416329Z"
    }
   },
   "outputs": [],
   "source": [
    "class GreedyPolicy(GamePolicy):\n",
    "    \"\"\"\n",
    "    A policy playing always one of the strongest turns.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, epsilon: float = 1):\n",
    "        _ = epsilon\n",
    "        super().__init__(1)\n",
    "\n",
    "    @property\n",
    "    def policy_name(self) -> str:\n",
    "        return \"greedy_policy\"\n",
    "\n",
    "    def _internal_policy(self, boards: np.ndarray) -> np.ndarray:\n",
    "        policies = np.random.rand(*boards.shape)\n",
    "        poss_turns = boards == 0  # checks where fields are empty.\n",
    "        poss_turns &= binary_dilation(boards == -1, SURROUNDING)\n",
    "        for game, idx, idy in itertools.product(\n",
    "            range(boards.shape[0]), range(BOARD_SIZE), range(BOARD_SIZE)\n",
    "        ):\n",
    "\n",
    "            if poss_turns[game, idx, idy]:\n",
    "                position = idx, idy\n",
    "                policies[game, idx, idy] += np.sum(\n",
    "                    np.array(\n",
    "                        list(\n",
    "                            _recursive_steps(boards[game, :, :], direction, position)\n",
    "                            for direction in DIRECTIONS\n",
    "                        )\n",
    "                    )\n",
    "                )\n",
    "        return policies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Putting the game simulation together\n",
    "Now it's time to bring all together for a proper simulation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Playing a single turn\n",
    "\n",
    "The next function needed is used to request a policy, verify that the turn is legit and place a stone and turn enemy stones if possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:42.564324Z",
     "start_time": "2023-03-30T23:51:42.448649Z"
    }
   },
   "outputs": [],
   "source": [
    "def single_turn(\n",
    "    current_boards: np, policy: GamePolicy\n",
    ") -> tuple[np.ndarray, np.ndarray]:\n",
    "    \"\"\"Execute a single turn on a board.\n",
    "\n",
    "    Places a new stone on the board. Turns captured enemy stones.\n",
    "\n",
    "    Args:\n",
    "        current_boards: The current board before the game.\n",
    "        policy: The game policy to be used.\n",
    "\n",
    "    Returns:\n",
    "        The new game board and the policy vector containing the index of the action used.\n",
    "    \"\"\"\n",
    "    policy_results = policy.get_policy(current_boards)\n",
    "\n",
    "    # if the constant VERIFY_POLICY is set to true the policy is verified. Should be good though.\n",
    "    # todo deactivate the policy verification after some testing.\n",
    "    if VERIFY_POLICY:\n",
    "        assert np.all(moves_possible(current_boards, policy_results)), (\n",
    "            current_boards[(moves_possible(current_boards, policy_results) == False)],\n",
    "            policy_results[(moves_possible(current_boards, policy_results) == False)],\n",
    "            np.where(moves_possible(current_boards, policy_results) == False),\n",
    "        )\n",
    "    return do_moves(current_boards, policy_results), policy_results"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "%timeit single_turn(get_new_games(EXAMPLE_STACK_SIZE), RandomPolicy(1))\n",
    "VERIFY_POLICY = False  # type: ignore\n",
    "%timeit single_turn(get_new_games(EXAMPLE_STACK_SIZE), RandomPolicy(1))\n",
    "VERIFY_POLICY = True  # type: ignore\n",
    "_turn_result = single_turn(get_new_games(EXAMPLE_STACK_SIZE), RandomPolicy(1))\n",
    "plot_othello_boards(_turn_result[0][:8], _turn_result[1][:8])\n",
    "del _turn_result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simulate a stack of games\n",
    "This function will simulate a stack of games and return an array of policies and histories.\n",
    "\n",
    "This will return an arrays with the size of (70 x n x 8 x 8) and (70 x n x 2).\n",
    "The first will contain the boards. The second will contain the actions. If no action is taken the action will be noted as played in (-1, -1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-30T23:51:42.565331Z",
     "start_time": "2023-03-30T23:51:42.461638Z"
    }
   },
   "outputs": [],
   "source": [
    "def simulate_game(\n",
    "    nr_of_games: int,\n",
    "    policies: tuple[GamePolicy, GamePolicy],\n",
    "    tqdm_on: bool = False,\n",
    ") -> tuple[np.ndarray, np.ndarray]:\n",
    "    \"\"\"Simulates a stack of games.\n",
    "\n",
    "    Args:\n",
    "        nr_of_games: The number of games that should be simulated.\n",
    "        policies: The policies that should be used to simulate the game.\n",
    "        tqdm_on: Switches tqdm on.\n",
    "\n",
    "    Returns:\n",
    "        A stack of board histories and actions.\n",
    "    \"\"\"\n",
    "    board_history_stack = np.zeros((SIMULATE_TURNS, nr_of_games, 8, 8), dtype=np.int8)\n",
    "    action_history_stack = np.zeros((SIMULATE_TURNS, nr_of_games, 2), dtype=np.int8)\n",
    "    current_boards = get_new_games(nr_of_games)\n",
    "    for turn_index in tqdm(range(SIMULATE_TURNS)) if tqdm_on else range(SIMULATE_TURNS):\n",
    "        policy_index = turn_index % 2\n",
    "        policy = policies[policy_index]\n",
    "        board_history_stack[turn_index, :, :, :] = current_boards\n",
    "        if policy_index == 1:\n",
    "            current_boards *= -1\n",
    "        current_boards, action_taken = single_turn(current_boards, policy)\n",
    "        action_history_stack[turn_index, :] = action_taken\n",
    "\n",
    "        if policy_index == 1:\n",
    "            current_boards *= -1\n",
    "\n",
    "    return board_history_stack, action_history_stack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing to do now is try out how the player act."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Simulating games\n",
    "\n",
    "Since now a simulator, a tool for visualisation and two policies exist a few games need to be simulated to verify proper function off all three elements.\n",
    "\n",
    "### Random vs. random policy\n",
    "First there is a simulation of a game between two random polices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 568392 characters]\n",
      "text/plain": [
       "<Figure size 1200x4800 with 61 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(0)\n",
    "simulation_results = simulate_game(1, (RandomPolicy(1), RandomPolicy(1)))\n",
    "_unique_bords, _unique_actions = drop_duplicate_boards(\n",
    "    simulation_results[0].reshape(-1, 8, 8), simulation_results[1].reshape(-1, 2)\n",
    ")\n",
    "plot_othello_boards(_unique_bords, actions=_unique_actions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "peak memory: 342.61 MiB, increment: 1.22 MiB\n",
      "11.7 s ± 448 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%memit simulate_game(100, (RandomPolicy(1), RandomPolicy(1)))\n",
    "%timeit simulate_game(100, (RandomPolicy(1), RandomPolicy(1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Greedy vs. greedy policy\n",
    "Then there is a simulation of a game between two greedy policies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 579584 characters]\n",
      "text/plain": [
       "<Figure size 1200x4800 with 61 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(1)\n",
    "simulation_results = simulate_game(1, (GreedyPolicy(1), GreedyPolicy(1)))\n",
    "_unique_bords, _unique_actions = drop_duplicate_boards(\n",
    "    simulation_results[0].reshape(-1, 8, 8), simulation_results[1].reshape(-1, 2)\n",
    ")\n",
    "plot_othello_boards(_unique_bords, actions=_unique_actions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "peak memory: 349.14 MiB, increment: 0.74 MiB\n",
      "29.4 s ± 1.64 s per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%memit simulate_game(100, (GreedyPolicy(1), GreedyPolicy(1)))\n",
    "%timeit simulate_game(100, (GreedyPolicy(1), GreedyPolicy(1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Random vs. greedy policy\n",
    "\n",
    "Last there was a simulation between random and greedy policy. Random playing as black and the greedy as white."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 571596 characters]\n",
      "text/plain": [
       "<Figure size 1200x4800 with 61 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(2)\n",
    "simulation_results = simulate_game(1, (RandomPolicy(1), GreedyPolicy(1)))\n",
    "_unique_bords, _unique_actions = drop_duplicate_boards(\n",
    "    simulation_results[0].reshape(-1, 8, 8), simulation_results[1].reshape(-1, 2)\n",
    ")\n",
    "plot_othello_boards(_unique_bords, actions=_unique_actions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Greedy vs. random policy\n",
    "\n",
    "Last there was a simulation between the greedy policy as black and the random policy as white."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 556776 characters]\n",
      "text/plain": [
       "<Figure size 1200x4800 with 61 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.random.seed(3)\n",
    "simulation_results = simulate_game(1, (GreedyPolicy(1), RandomPolicy(1)))\n",
    "_unique_bords, _unique_actions = drop_duplicate_boards(\n",
    "    simulation_results[0].reshape(-1, 8, 8), simulation_results[1].reshape(-1, 2)\n",
    ")\n",
    "plot_othello_boards(_unique_bords, actions=_unique_actions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistical examination of the natural action space and game result\n",
    "As for many project some evaluation of the project is in order.\n",
    "\n",
    "1. What is the expected distribution of scores\n",
    "2. What is the expected distribution of possible actions\n",
    "\n",
    "    a. over time\n",
    "    \n",
    "    b. ober space\n",
    "\n",
    "The easiest and robustest way to analyse this is when analyzing randomly played games."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this purpose we played a sample of 10,000 games and saved them for later analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((70, 10000, 8, 8), (70, 10000, 2))"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def load_board_history():\n",
    "    if not os.path.exists(\"rnd_history.npy\") and not os.path.exists(\"rnd_action.npy\"):\n",
    "        simulation_results = simulate_game(\n",
    "            10_000, (RandomPolicy(1), RandomPolicy(1)), tqdm_on=True\n",
    "        )\n",
    "        board_history, action_history = simulation_results\n",
    "        np.save(\"rnd_history.npy\", board_history.astype(np.int8))\n",
    "        np.save(\"rnd_action.npy\", action_history.astype(np.int8))\n",
    "    else:\n",
    "        board_history = np.load(\"rnd_history.npy\")\n",
    "        action_history = np.load(\"rnd_action.npy\")\n",
    "    return board_history, action_history\n",
    "\n",
    "\n",
    "_board_history, _action_history = load_board_history()\n",
    "_board_history.shape, _action_history.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For those 10k games the possible actions where evaluated and saved for each and every turn in the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((70, 10000, 8, 8), (70, 10000, 2))"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "if not os.path.exists(\"turn_possible.npy\"):\n",
    "    __board_history = _board_history.copy()\n",
    "    __board_history[1::2] = __board_history[1::2] * -1\n",
    "\n",
    "    _poss_turns = get_possible_turns(\n",
    "        __board_history.reshape((-1, 8, 8)), tqdm_on=True\n",
    "    ).reshape((SIMULATE_TURNS, -1, 8, 8))\n",
    "    np.save(\"turn_possible.npy\", _poss_turns)\n",
    "    del __board_history\n",
    "_poss_turns = np.load(\"turn_possible.npy\")\n",
    "_poss_turns.shape, _action_history.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Those possible turms then where counted for all games in the history stack.\n",
    "\n",
    "### Action space over time / tree search size estimation\n",
    "The action space size can be drawn into a histogram by turn and a curve over the mean action space size.\n",
    "This can be used to analyse in which area of the game that cant be solved absolutely."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "24d55452a53240a0807d1f396bdd4632",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=34, description='turn', max=69), Output()), _dom_classes=('widget-intera…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "count_poss_turns = np.sum(_poss_turns, axis=(2, 3))\n",
    "mean_possibility_count = np.mean(count_poss_turns, axis=1)\n",
    "std_possibility_count = np.std(count_poss_turns, axis=1)\n",
    "cum_prod = count_poss_turns\n",
    "\n",
    "\n",
    "@interact(turn=(0, 69))\n",
    "def poss_turn_count(turn):\n",
    "    fig, axes = plt.subplots(2, 2, figsize=(15, 8))\n",
    "    ax1, ax2, ax3, ax4 = axes.flatten()\n",
    "    _mean_possibility_count = mean_possibility_count.copy()\n",
    "    _std_possibility_count = std_possibility_count.copy()\n",
    "    _mean_possibility_count[_mean_possibility_count <= 1] = 1\n",
    "    _std_possibility_count[_std_possibility_count <= 1] = 1\n",
    "    # np.cumprod(_mean_possibility_count[::-1], axis=0)[::-1]\n",
    "    # todo what happens here=\n",
    "    fig.suptitle(\n",
    "        f\"Action space size analysis\\nThe total size is estimated to be around {np.prod(_mean_possibility_count):.2E}\"\n",
    "    )\n",
    "    ax1.hist(count_poss_turns[turn], density=True)\n",
    "    ax1.set_title(f\"Histogram of the action space size for turn {turn}\")\n",
    "    ax1.set_xlabel(\"Action space size\")\n",
    "    ax1.set_ylabel(\"Action space size probability\")\n",
    "    ax2.set_title(f\"Mean size of the action space per turn\")\n",
    "    ax2.set_xlabel(\"Turn\")\n",
    "    ax2.set_ylabel(\"Average possible moves\")\n",
    "\n",
    "    ax2.errorbar(\n",
    "        range(70),\n",
    "        mean_possibility_count,\n",
    "        yerr=std_possibility_count,\n",
    "        label=\"Mean action space size with error bars\",\n",
    "    )\n",
    "    ax2.scatter(turn, mean_possibility_count[turn], marker=\"x\")\n",
    "    ax2.legend()\n",
    "\n",
    "    action_space_cumprod = np.cumprod(_mean_possibility_count[::-1], axis=0)[::-1]\n",
    "    ax4.plot(range(70), action_space_cumprod)\n",
    "\n",
    "    ax4.scatter(turn, action_space_cumprod[turn], marker=\"x\")\n",
    "    ax4.set_yscale(\"log\", base=10)\n",
    "    ax4.set_xlabel(\"Turn\")\n",
    "    ax4.set_ylabel(\"Mean remaining total action space size\")\n",
    "    ax4.set_title(\n",
    "        f\"Remaining action space at {turn} = {action_space_cumprod[turn].round():.2E}\"\n",
    "    )\n",
    "    fig.delaxes(ax3)\n",
    "    fig.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The analysis of the action space can be used to switch between a \"normal\" algorithm and an ANN-powered algorithm, depending on the network's performance and the speed requirements. If the average remaining decision tree is small enough, this can be done after step 52, but it could also start earlier, such as at step 50 or 48 if desired."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is interesting to see that the action space for the first player (black) is much smaller than for the second player. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Total mean action-space</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>white</th>\n",
       "      <td>3.753117e+20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>black</th>\n",
       "      <td>5.687159e+18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Total mean action-space\n",
       "white             3.753117e+20\n",
       "black             5.687159e+18"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "black = mean_possibility_count[0::2]\n",
    "white = mean_possibility_count[1::2]\n",
    "df = pd.DataFrame(\n",
    "    [\n",
    "        {\n",
    "            \"white\": np.prod(np.extract(white, white)),\n",
    "            \"black\": np.prod(np.extract(black, black)),\n",
    "        }\n",
    "    ],\n",
    "    index=[\"Total mean action-space\"],\n",
    ").T\n",
    "del white, black\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Possibility that an action can be taken at a specifc turn\n",
    "\n",
    "The following diagram displays the available positions for placing a stone at each turn. This can help compare the learning behavior of different policies, particularly with regard to the corners. A low probability for a corner suggests that the AI avoids giving corners to the opponent and tries to capture them for itself whenever possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4f7639d8bbc644d0881612681bbeba56",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=34, description='turn', max=69), Output()), _dom_classes=('widget-intera…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mean_poss_turn = np.mean(_poss_turns, axis=1)\n",
    "\n",
    "\n",
    "@interact(turn=(0, 69))\n",
    "def turn_distribution_heatmap(turn):\n",
    "    turn_possibility_on_field = mean_poss_turn[turn]\n",
    "\n",
    "    sns.heatmap(\n",
    "        turn_possibility_on_field,\n",
    "        linewidth=0.5,\n",
    "        square=True,\n",
    "        annot=True,\n",
    "        xticklabels=\"ABCDEFGH\",\n",
    "        yticklabels=list(range(1, 9)),\n",
    "    )\n",
    "    plt.title(f\"Headmap of where stones can be placed on turn {turn}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Statistic of skipped actions\n",
    "\n",
    "Not all turns can be played. Plotted as a mean over the curse of the game it can be clearly seen that the first time a turn can be skipped is turn 9 and increases over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 29884 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def history_changed(board_history: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"Calculates if the board changed between actions.\n",
    "\n",
    "    Args:\n",
    "        board_history: A history of game boards. Shaped (70 * n * 8 * 8)\n",
    "    \"\"\"\n",
    "    return ~np.all(\n",
    "        np.roll(board_history, shift=1, axis=0) == board_history, axis=(2, 3)\n",
    "    )\n",
    "\n",
    "\n",
    "plt.title(\"Share of turns skipped\")\n",
    "plt.plot(1 - np.mean(history_changed(_board_history), axis=1))\n",
    "plt.xlabel(\"Turn\")\n",
    "plt.ylabel(\"Factor of skipped turns\")\n",
    "plt.yscale(\"log\", base=10)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Hash branching\n",
    "To calibrate the exploration rate properly we compared all the games in a stack of games. The graph shows the number of unique game boards at each of the game turns.\n",
    "As can be seen below for random games the games start to be unique very fast.\n",
    "For a proper directed exploration I assume the rate needs to be calibrated that the game still have some duplications of the best knwon game at the end of a game simulation left."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 23660 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def calculate_board_branching(board_history) -> pd.Series:\n",
    "    assert len(board_history.shape) == 4\n",
    "    assert board_history.shape[-2:] == (8, 8)\n",
    "    assert board_history.shape[0] == SIMULATE_TURNS\n",
    "    return pd.Series(\n",
    "        [count_unique_boards(board_history[turn]) for turn in range(SIMULATE_TURNS)]\n",
    "    )\n",
    "\n",
    "\n",
    "_ = calculate_board_branching(_board_history).plot(\n",
    "    title=f\"Exploration history over {_board_history.shape[0]} turns\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 3.0000e+00 -3.0000e+00  3.2566e+00 -3.4776e+00  3.4422e+00 -3.8032e+00\n",
      "  3.7130e+00 -3.9864e+00  3.9188e+00 -4.1371e+00  4.0475e+00 -4.2540e+00\n",
      "  4.1889e+00 -4.4112e+00  4.3339e+00 -4.4949e+00  4.5001e+00 -4.5824e+00\n",
      "  4.5818e+00 -4.7497e+00  4.7366e+00 -4.8246e+00  4.8160e+00 -4.9348e+00\n",
      "  4.9494e+00 -5.0466e+00  5.1068e+00 -5.1586e+00  5.2366e+00 -5.2766e+00\n",
      "  5.3122e+00 -5.4081e+00  5.4527e+00 -5.5603e+00  5.5967e+00 -5.6423e+00\n",
      "  5.7014e+00 -5.8357e+00  5.8643e+00 -5.9012e+00  5.9900e+00 -6.1215e+00\n",
      "  6.1590e+00 -6.2150e+00  6.3472e+00 -6.4288e+00  6.5409e+00 -6.6730e+00\n",
      "  6.7297e+00 -6.9038e+00  6.9386e+00 -7.0992e+00  7.2869e+00 -7.4165e+00\n",
      "  7.5312e+00 -7.7480e+00  7.9052e+00 -7.9656e+00  7.8611e+00 -7.3040e+00\n",
      "  3.0707e+00 -7.1540e-01  1.4300e-01 -1.9100e-02  3.4000e-03 -3.0000e-04\n",
      "  5.0000e-04  0.0000e+00  0.0000e+00  0.0000e+00]\n"
     ]
    }
   ],
   "source": [
    "_direct_score = np.sum(_board_history, axis=(-2, -1))\n",
    "_score = np.zeros_like(_direct_score)\n",
    "_score[:-1] = _direct_score[1:] - _direct_score[:-1]\n",
    "print(np.mean(_score, axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reword functions\n",
    "\n",
    "For any kind of reinforcement learning is a reword function needed.\n",
    "For otello this would be the final score, the information who won, changes to the score and the sum of the board.\n",
    "A combination of those three also be possible.\n",
    "It is probably not be possible to weight the current score to high in a reword function since that would be to close to a classic greedy algorithm.\n",
    "But some direct influence would increase the learning speed.\n",
    "In the next section are all three reword functions implemented to be combined and weight later on as needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate the final game score\n",
    "\n",
    "In Otello, the empty fields at the end of the game are counted for the player with more stones. The function below calculates this and returns the score difference for player 1 (black)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "411 µs ± 88.9 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)\n"
     ]
    }
   ],
   "source": [
    "def final_boards_evaluation(boards: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"Evaluates the board at the end of the game.\n",
    "\n",
    "    All unused fields are added to the score of the player that has more stones with his color up.\n",
    "    This score only applies to the end of the game.\n",
    "    Normally the score is represented by the number of stones each player has.\n",
    "    In this case the score was combined by building the difference.\n",
    "\n",
    "    Args:\n",
    "        boards: A stack of game bords ot the end of the game. Shaped (n * 8 * 8)\n",
    "\n",
    "    Returns:\n",
    "        the combined score for both player.\n",
    "    \"\"\"\n",
    "    score1, score2 = np.sum(boards == 1, axis=(1, 2)), np.sum(boards == -1, axis=(1, 2))\n",
    "    player_1_won = score1 > score2\n",
    "    player_2_won = score1 < score2\n",
    "    score1_final = 64 - score2[player_1_won] * 2\n",
    "    score2_final = 64 - score1[player_2_won] * 2\n",
    "    score = np.zeros(boards.shape[0])\n",
    "    score[player_1_won] = score1_final\n",
    "    score[player_2_won] = -score2_final\n",
    "    return score\n",
    "\n",
    "\n",
    "np.random.seed(2)\n",
    "_boards = simulate_game(10, (RandomPolicy(1), RandomPolicy(1)))[0]\n",
    "np.testing.assert_array_equal(\n",
    "    np.sum(_boards[-1], axis=(1, 2)), final_boards_evaluation(_boards[-1])\n",
    ")\n",
    "np.random.seed(2)\n",
    "np.testing.assert_array_equal(\n",
    "    np.array([-6.0, -36.0, -12.0, -16.0, 38.0, -12.0, 2.0, -22.0, 2.0, 10.0]),\n",
    "    final_boards_evaluation(\n",
    "        simulate_game(10, (RandomPolicy(1), RandomPolicy(1)))[0][-1]\n",
    "    ),\n",
    ")\n",
    "\n",
    "np.random.seed(2)\n",
    "boards = simulate_game(10, (RandomPolicy(1), RandomPolicy(1)))[0][-1]\n",
    "boards[:, 4, :] = 0\n",
    "np.testing.assert_array_equal(\n",
    "    np.array([-14.0, -38.0, -14.0, -22.0, 40.0, -16.0, -14.0, -28.0, 0.0, 20.0]),\n",
    "    final_boards_evaluation(boards),\n",
    ")\n",
    "\n",
    "_boards = get_new_games(EXAMPLE_STACK_SIZE)\n",
    "%timeit final_boards_evaluation(_boards)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def calculate_final_evaluation_for_history(board_history: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"Calculates the final scores for a stack of game histories.\n",
    "\n",
    "    Args:\n",
    "        board_history: A stack of game histories.\n",
    "    \"\"\"\n",
    "    final_evaluation = final_boards_evaluation(board_history[-1])\n",
    "    return final_evaluation / 64\n",
    "\n",
    "\n",
    "np.random.seed(2)\n",
    "_boards = simulate_game(10, (RandomPolicy(1), RandomPolicy(1)))[0]\n",
    "np.testing.assert_array_equal(\n",
    "    np.array([-6.0, -36.0, -12.0, -16.0, 38.0, -12.0, 2.0, -22.0, 2.0, 10.0]) / 64,\n",
    "    calculate_final_evaluation_for_history(_boards),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 25964 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assert len(calculate_final_evaluation_for_history(_board_history).shape) == 1\n",
    "_final_eval = calculate_final_evaluation_for_history(_board_history)\n",
    "plt.title(\"Histogram over the final score distribution\")\n",
    "plt.hist((_final_eval * 64), density=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Evaluation game by stones only\n",
    "\n",
    "The next evaluation is just by counting stones by color and building the difference between both. In this implementation it can also be called the sum of a board."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "59.5 µs ± 11.2 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)\n"
     ]
    }
   ],
   "source": [
    "def evaluate_boards(boards: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"Counts the stones each player has on the board.\n",
    "\n",
    "    Args:\n",
    "        boards: A stack of boards for evaluation. Shaped (n * 8 * 8)\n",
    "\n",
    "    Returns:\n",
    "        the combined score for both players.\n",
    "    \"\"\"\n",
    "    assert boards.shape[-2:] == (8, 8)\n",
    "    return np.sum(boards, axis=(-1, -2))\n",
    "\n",
    "\n",
    "np.random.seed(1)\n",
    "np.testing.assert_array_equal(\n",
    "    evaluate_boards(simulate_game(10, (RandomPolicy(1), RandomPolicy(1)))[0][-1]),\n",
    "    np.array([-30, -14, -8, 4, -4, -8, -36, 14, -16, -4]),\n",
    ")\n",
    "np.random.seed(2)\n",
    "np.testing.assert_array_equal(\n",
    "    evaluate_boards(simulate_game(10, (RandomPolicy(1), RandomPolicy(1)))[0][-1]),\n",
    "    np.array([-6, -36, -12, -16, 38, -12, 2, -22, 2, 10]),\n",
    ")\n",
    "np.testing.assert_array_equal(\n",
    "    evaluate_boards(simulate_game(10, (RandomPolicy(1), RandomPolicy(1)))[0]).shape,\n",
    "    (70, 10),\n",
    ")\n",
    "np.random.seed(3)\n",
    "np.testing.assert_array_equal(\n",
    "    evaluate_boards(simulate_game(10, (RandomPolicy(1), RandomPolicy(1)))[0][:4, :3]),\n",
    "    np.array([[0, 0, 0], [3, 3, 3], [0, 0, 0], [5, 3, 3]]),\n",
    ")\n",
    "\n",
    "_boards = get_new_games(EXAMPLE_STACK_SIZE)\n",
    "%timeit evaluate_boards(_boards)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 19508 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "_eval = evaluate_boards(_board_history[-1])\n",
    "plt.title(\"Histogram over the final score distribution\")\n",
    "plt.hist(_eval, density=True)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate the winner of a game\n",
    "\n",
    "The last function evaluates who won by calculating who signum function of the sum of the numpy array representing the board.\n",
    "The resulting number would be one of the game was won by the player (white) or -1 if the enemy (black) won. The result would also"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "126 µs ± 11.3 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)\n"
     ]
    }
   ],
   "source": [
    "def calculate_who_won(board_history: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"Checks who won or is winning a game.\n",
    "\n",
    "    Args:\n",
    "        board_history: A stack of boards for evaluation. Shaped (70 * n * 8 * 8)\n",
    "\n",
    "    Returns:\n",
    "        The information who won for both player. 1 meaning the player won, -1 means the opponent lost. 0 represents a patt.\n",
    "    \"\"\"\n",
    "    assert board_history.shape[-2:] == (8, 8)\n",
    "    assert board_history.shape[0] == 70\n",
    "    return np.sign(np.sum(board_history[-1], axis=(1, 2)))\n",
    "\n",
    "\n",
    "np.random.seed(1)\n",
    "np.testing.assert_array_equal(\n",
    "    calculate_who_won(simulate_game(10, (RandomPolicy(1), RandomPolicy(1)))[0]),\n",
    "    np.array([-1, -1, -1, 1, -1, -1, -1, 1, -1, -1]),\n",
    ")\n",
    "np.random.seed(2)\n",
    "np.testing.assert_array_equal(\n",
    "    calculate_who_won(simulate_game(10, (RandomPolicy(1), RandomPolicy(1)))[0]),\n",
    "    np.array([-1, -1, -1, -1, 1, -1, 1, -1, 1, 1]),\n",
    ")\n",
    "\n",
    "\n",
    "_boards = simulate_game(EXAMPLE_STACK_SIZE, (RandomPolicy(1), RandomPolicy(1)))[0]\n",
    "%timeit calculate_who_won(_boards)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 13536 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title(\"Win distribution\")\n",
    "plt.bar(\n",
    "    [\"black\", \"draw\", \"white\"],\n",
    "    pd.Series(calculate_who_won(_board_history)).value_counts().sort_index()\n",
    "    / _board_history.shape[1],\n",
    ")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Direct turn evaluation\n",
    "\n",
    "Besides evaluating the turn there is always the possibility to calculate how much of an direct impact a single turn had."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_direct_score(board_history: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"Calculates the delta score for all actions.\n",
    "\n",
    "    Args:\n",
    "        board_history: A history of board games or a stack of board games.  Shaped (70 * n * 8 * 8)\n",
    "    \"\"\"\n",
    "    assert board_history.shape[0] == 70\n",
    "    assert board_history.shape[-2:] == (8, 8)\n",
    "    direct_score = np.sum(board_history, axis=(-2, -1))\n",
    "    score = np.zeros_like(direct_score)\n",
    "    score[:-1] = direct_score[1:] - direct_score[:-1]\n",
    "    return score / 64\n",
    "\n",
    "\n",
    "assert len(calculate_direct_score(_board_history).shape) == 2\n",
    "assert calculate_direct_score(_board_history).shape[0] == SIMULATE_TURNS\n",
    "np.random.seed(2)\n",
    "np.testing.assert_equal(\n",
    "    calculate_direct_score(simulate_game(10, (RandomPolicy(1), RandomPolicy(1)))[0])[\n",
    "        :, 0\n",
    "    ][:10]\n",
    "    * 64,\n",
    "    np.array(\n",
    "        [3.0, -3.0, 3.0, -5.0, 5.0, -5.0, 3.0, -3.0, 7.0, -5.0],\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When plotting the direct score it can be easily seen that the later turnse are point-wise more important. A bad opening however will not allow the player to keep those points. But it is easy to see that points not made at the beginning of the game can be made at the end of the game. This allows for concentration on the gameplay and some preparation at the start of the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "204b87e308144d63952e28424efe8fac",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=29, description='turn', max=59), Output()), _dom_classes=('widget-intera…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "score_history = calculate_direct_score(_board_history)\n",
    "score_history *= 64\n",
    "\n",
    "\n",
    "@interact(turn=(0, 59))\n",
    "def hist_direct_score(turn):\n",
    "    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6))\n",
    "    fig.suptitle(\n",
    "        f\"Action space size analysis / total size estimate {np.prod(np.extract(mean_possibility_count, mean_possibility_count)):.4g}\"\n",
    "    )\n",
    "\n",
    "    ax1.set_title(\n",
    "        f\"Histogram of scores changes on turn {turn} by {'white' if turn % 2 == 0 else 'black'}\"\n",
    "    )\n",
    "    score = score_history[turn]\n",
    "    bins = max(1, int(max(score) - min(score)))\n",
    "    ax1.hist(score, density=True, bins=bins)\n",
    "    ax1.set_xlabel(\"Points made\")\n",
    "    ax1.set_ylabel(\"Score probability\")\n",
    "    ax2.set_title(\"Points scored at turn\")\n",
    "    ax2.set_xlabel(\"Turn\")\n",
    "    ax2.set_ylabel(\"Average points scored\")\n",
    "    ax2.errorbar(\n",
    "        range(60),\n",
    "        np.abs(np.mean(score_history, axis=1)[:60]),\n",
    "        yerr=np.std(score_history, axis=1)[:60],\n",
    "        label=\"Mean score at turn\",\n",
    "    )\n",
    "    ax2.scatter(\n",
    "        turn, np.abs(np.mean(score_history, axis=1))[:60][turn], marker=\"x\", color=\"red\"\n",
    "    )\n",
    "    ax2.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Q-Learning Policies\n",
    "Q-learning is a classic reinforcement learning technique. The Q-function is an action-value function that returns the expected value of an action in a given state.\n",
    "\n",
    "$Q^\\pi(s_t,a_t)=\\sum^{60}_{t=turn}\\gamma^{60-t} \\cdot R_t$\n",
    "\n",
    "With this function, all actions in a given state can be evaluated, and the most beneficial action can be taken. With classical reinforcement learning, a table for situations and actions is explored and slowly filled. With ANNs, there is the possibility to use an AI model that can interpolate between situations and should not need to explore the complete game tree to solve some situations.\n",
    "\n",
    "### Calculating discount tables\n",
    "\n",
    "Since the game stack contains all steps, even if no action is possible, this needs to be corrected. The normal formula for a reward is:\n",
    "\n",
    "$E(s_{turn},a_{turn}) = \\sum^{60}_{t=turn}\\gamma^{60-t} \\cdot R_t$\n",
    "\n",
    "Since turns that can't be taken do not have the element of uncertainty, the discounting has to be excluded by setting the value to $1$ instead of $\\gamma$.\n",
    "\n",
    "$\\gamma^*_t =\\begin{cases}1 & |a_t|=0\\\\gamma & |a_t|>0\\end{cases}$\n",
    "\n",
    "$E(s_{turn},a_{turn}) = \\prod_{t=turn}^{70}\\gamma^*_t \\cdot R_t$\n",
    "\n",
    "The table below contains the aggregated discount factors ($\\prod_{t=turn}^{70}\\gamma^*_t$) for each reward fitting to the state history. This setup also allows to reward the certainty gained by taking the choice of the action from the opponent. It can be argued that also all turns where a player had no choice how to act should not be discounted. But this will increase calculation requirements to nearly double, which is currently not acceptable since computation time and code complexity are bottlenecks.\n",
    "Since at least 10 turns in the stack ar not taken boat policies should lead to ruffly equal results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_gamma_table(board_history: np.ndarray, gamma_value: float) -> np.ndarray:\n",
    "    \"\"\"Calculates a discount table for a board history.\n",
    "\n",
    "    Args:\n",
    "        board_history: A history of game boards. Shaped (70 * n * 8 * 8)\n",
    "        gamma_value: The default discount factor.\n",
    "    \"\"\"\n",
    "    unchanged = history_changed(board_history)\n",
    "    gamma_values = np.ones_like(unchanged, dtype=float)\n",
    "    gamma_values[unchanged] = gamma_value\n",
    "    return gamma_values\n",
    "\n",
    "\n",
    "assert get_gamma_table(_board_history, 0.8).shape == _board_history.shape[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calculating rewords\n",
    "\n",
    "To calculate the rewards for the Reversi AI, we will use the $\\gamma$ values to combine the rewards obtained during the game with the functions calculate_direct_score, calculate_final_evaluation_for_history, and calculate_who_won.\n",
    "\n",
    "The rewards obtained will be used to build a weighted sum of rewards, where most of the rewards are terminal rewards awarded at the end of the game and discounted back over the course of the game. The sum of the reward weights is always 1, with the third value calculated from the first two.\n",
    "\n",
    "The direct score is the only part of the reward that is awarded before the terminal reward. This setup allows for experimentation with different types of rewards to train the model, with different definitions of what is considered \"best\" depending on factors such as initial startup time, stability, and quality of results.\n",
    "\n",
    "Although $Q^\\pi$ depends on state and action, the rewards are returned and do not require a specified action to be given, since the action is implied by the structure of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "def calculate_q_reword(\n",
    "    board_history: np.ndarray,\n",
    "    who_won_fraction: float = 0.2,\n",
    "    final_score_fraction: float = 0.2,\n",
    "    gamma: float = 0.8,\n",
    ") -> np.ndarray:\n",
    "    \"\"\"Calculates a Q reword for a stack of states.\n",
    "\n",
    "    Args:\n",
    "        board_history: A stack ob board histories to calculate q_rewords for.\n",
    "        who_won_fraction: This factor describes how the winner of the game should be weighted. Expected value is in [0, 1].\n",
    "        final_score_fraction: This factor describes how important the final score of the game should be weighted. Expected value is in [0, 1].\n",
    "        gamma: The discount value fo all turns that had a choice.\n",
    "    \"\"\"\n",
    "    assert who_won_fraction + final_score_fraction <= 1\n",
    "    assert final_score_fraction >= 0\n",
    "    assert who_won_fraction >= 0\n",
    "\n",
    "    gama_table = get_gamma_table(board_history, gamma)\n",
    "    combined_score = np.zeros_like(gama_table)\n",
    "    combined_score += calculate_direct_score(board_history) * (\n",
    "        1 - (who_won_fraction + final_score_fraction)\n",
    "    )\n",
    "    combined_score[-1] += (\n",
    "        calculate_final_evaluation_for_history(board_history)\n",
    "        * final_score_fraction\n",
    "        / 0.7\n",
    "    )\n",
    "    combined_score[-1] += calculate_who_won(board_history) * who_won_fraction\n",
    "    for turn in range(SIMULATE_TURNS - 1, 0, -1):\n",
    "        values = gama_table[turn] * combined_score[turn]\n",
    "        combined_score[turn - 1] += values\n",
    "\n",
    "    return combined_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The calculated q_learning rewords look than as shown below. For the different distributions of rewords by factor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-9.07181894e-11, -3.85552305e-10,  6.80386421e-11, ...,\n",
       "         9.07181894e-11,  0.00000000e+00,  1.13397737e-10],\n",
       "       [-1.29597413e-10, -5.50789007e-10,  9.71980601e-11, ...,\n",
       "         1.29597413e-10,  0.00000000e+00,  1.61996767e-10],\n",
       "       [-1.85139162e-10, -7.86841439e-10,  1.38854372e-10, ...,\n",
       "         1.85139162e-10,  0.00000000e+00,  2.31423953e-10],\n",
       "       ...,\n",
       "       [-1.78571429e-01, -7.58928571e-01,  1.33928571e-01, ...,\n",
       "         1.78571429e-01,  0.00000000e+00,  2.23214286e-01],\n",
       "       [-1.78571429e-01, -7.58928571e-01,  1.33928571e-01, ...,\n",
       "         1.78571429e-01,  0.00000000e+00,  2.23214286e-01],\n",
       "       [-1.78571429e-01, -7.58928571e-01,  1.33928571e-01, ...,\n",
       "         1.78571429e-01,  0.00000000e+00,  2.23214286e-01]])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calculate_q_reword(\n",
    "    _board_history, gamma=0.7, who_won_fraction=0, final_score_fraction=1\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-1.53249554e-06, -1.91561943e-06, -2.39452428e-06, -2.99315535e-06,\n",
       "       -3.74144419e-06, -4.67680524e-06, -5.84600655e-06, -7.30750819e-06,\n",
       "       -9.13438523e-06, -1.14179815e-05, -1.42724769e-05, -1.78405962e-05,\n",
       "       -2.23007452e-05, -2.78759315e-05, -3.48449144e-05, -4.35561430e-05,\n",
       "       -5.44451787e-05, -6.80564734e-05, -8.50705917e-05, -1.06338240e-04,\n",
       "       -1.32922800e-04, -1.66153499e-04, -2.07691874e-04, -2.59614843e-04,\n",
       "       -3.24518554e-04, -4.05648192e-04, -5.07060240e-04, -6.33825300e-04,\n",
       "       -7.92281625e-04, -9.90352031e-04, -1.23794004e-03, -1.54742505e-03,\n",
       "       -1.93428131e-03, -2.41785164e-03, -3.02231455e-03, -3.77789319e-03,\n",
       "       -4.72236648e-03, -5.90295810e-03, -7.37869763e-03, -9.22337204e-03,\n",
       "       -1.15292150e-02, -1.44115188e-02, -1.80143985e-02, -2.25179981e-02,\n",
       "       -2.81474977e-02, -3.51843721e-02, -4.39804651e-02, -5.49755814e-02,\n",
       "       -6.87194767e-02, -8.58993459e-02, -1.07374182e-01, -1.34217728e-01,\n",
       "       -1.67772160e-01, -2.09715200e-01, -2.62144000e-01, -3.27680000e-01,\n",
       "       -4.09600000e-01, -5.12000000e-01, -6.40000000e-01, -6.40000000e-01,\n",
       "       -8.00000000e-01, -8.00000000e-01, -1.00000000e+00, -1.00000000e+00,\n",
       "       -1.00000000e+00, -1.00000000e+00, -1.00000000e+00, -1.00000000e+00,\n",
       "       -1.00000000e+00, -1.00000000e+00])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calculate_q_reword(\n",
    "    _board_history, gamma=0.8, who_won_fraction=1, final_score_fraction=0\n",
    ")[:, 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  1.53941436,  -1.82573204,   1.46783494,  -1.91520632,\n",
       "         1.3559921 ,  -4.55500988,   0.55623766,  -3.05470293,\n",
       "        -0.06837866,  -6.33547333,   0.83065834,  -2.71167707,\n",
       "         0.36040366,  -3.29949543,   2.12563071,  -3.59296161,\n",
       "         6.75879799,   4.69849749,   9.62312186,   5.77890232,\n",
       "        13.47362791,   8.09203488,  13.8650436 ,   1.0813045 ,\n",
       "         5.10163063,   0.12703829,   3.90879786,   1.13599733,\n",
       "         5.16999666,  -4.78750418,   0.26561978,  -3.41797528,\n",
       "         1.9775309 ,  -3.77808638,  -0.97260797,  -7.46575996,\n",
       "         1.91780005,  -1.35274994,   4.55906257,  -8.05117178,\n",
       "         1.18603527,  -4.76745591,   0.29068011,  -3.38664986,\n",
       "        -0.48331232,  -4.3541404 ,   3.3073245 ,   0.38415562,\n",
       "         4.23019452,   1.53774316,   8.17217894,   1.46522368,\n",
       "         5.5815296 ,  -4.273088  ,  -1.59136   , -15.7392    ,\n",
       "        -8.424     , -24.28      , -26.6       , -26.6       ,\n",
       "       -17.        , -17.        ,   0.        ,   0.        ,\n",
       "         0.        ,   0.        ,   0.        ,   0.        ,\n",
       "         0.        ,   0.        ])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calculate_q_reword(\n",
    "    _board_history, gamma=0.8, who_won_fraction=0, final_score_fraction=0\n",
    ")[:, 0] * 64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Layer Initialization Function\n",
    "\n",
    "When training the ANN, it is important to initialize the weights and biases of its layers before starting the training process. This can be done using various techniques such as random initialization or using pre-trained weights. The choice of initialization function can have an impact on the training speed and the quality of the learned model. Therefore, we will carefully select an appropriate initialization function to ensure the best possible training outcome for our Otello AI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def weights_init_normal(m):\n",
    "    \"\"\"Takes in a module and initializes all linear layers with weight\n",
    "       values taken from a normal distribution.\n",
    "    Source: https://stackoverflow.com/a/55546528/11003343\n",
    "    \"\"\"\n",
    "\n",
    "    classname = m.__class__.__name__\n",
    "    # for every Linear layer in a model\n",
    "    if classname.find(\"Linear\") != -1:\n",
    "        y = m.in_features\n",
    "        # m.weight.data should be taken from a normal distribution\n",
    "        m.weight.data.normal_(0.0, 1 / np.sqrt(y))\n",
    "        # m.bias.data should be 0\n",
    "        m.bias.data.fill_(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ANN Architecture\n",
    "\n",
    "The architecture of an Artificial Neural Network (ANN) is defined by its inputs and outputs. In the case of Q-Learning ANNs for the board game Otello, the network should calculate the expected reward for the current state and a proposed action.\n",
    "\n",
    "The state of the game board is represented by a 8x8 array, where each cell can be either empty, occupied by a black stone, or occupied by a white stone. The proposed action is represented by an array of equal size, using one-hot encoding to indicate the cell where a stone is proposed to be placed.\n",
    "\n",
    "The output of the ANN should be a value between -1 and 1, which suggests using a $\\tanh$ activation function. Since the game board is small and the position of the stones is crucial, it was decided to use only conventional layers, without any Long Short-Term Memory (LSTM) layer or other architectures for sequences and time series, as the history of the game does not matter.\n",
    "\n",
    "Therefore, the input size of the network is (8 x 8 x 2) and the output size is 1, representing the expected reward for the proposed action in the current state of the game board."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "class DQLNet(nn.Module):\n",
    "    \"\"\"A simple ANN structure\"\"\"\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(8 * 8 * 2, 128 * 2)\n",
    "        self.fc2 = nn.Linear(128 * 2, 128 * 3)\n",
    "        self.fc3 = nn.Linear(128 * 3, 128 * 2)\n",
    "        self.fc4 = nn.Linear(128 * 2, 1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        if isinstance(x, np.ndarray):\n",
    "            x = torch.from_numpy(x).float()\n",
    "        x = torch.flatten(x, 1)\n",
    "        x = self.fc1(x)\n",
    "        x = functional.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        x = functional.relu(x)\n",
    "        x = self.fc3(x)\n",
    "        x = functional.relu(x)\n",
    "        x = self.fc4(x)\n",
    "        x = torch.tanh(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since the first ANN was initially not very successful a reduced version was also tried."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "class DQLSimple(nn.Module):\n",
    "    \"\"\"A even simpler ANN structure\"\"\"\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(8 * 8 * 2, 64 * 3)\n",
    "        self.fc2 = nn.Linear(64 * 3, 128 * 2)\n",
    "        self.fc3 = nn.Linear(128 * 2, 1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        if isinstance(x, np.ndarray):\n",
    "            x = torch.from_numpy(x).float()\n",
    "        x = torch.flatten(x, 1)\n",
    "        x = self.fc1(x)\n",
    "        x = functional.relu(x)\n",
    "        x = self.fc2(x)\n",
    "        x = functional.relu(x)\n",
    "        x = self.fc3(x)\n",
    "        x = torch.tanh(x)\n",
    "        return x\n",
    "\n",
    "\n",
    "assert DQLNet().forward(np.zeros((5, 2, 8, 8))).shape == (5, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "class SymmetryMode(Enum):\n",
    "    MULTIPLY = \"MULTIPLY\"\n",
    "    BREAK_SEQUENCE = \"BREAK_SEQUENCE\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((70, 100, 8, 8), (70, 100, 2))"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_board_history, _action_history = simulate_game(100, (RandomPolicy(1), RandomPolicy(1)))\n",
    "_board_history.shape, _action_history.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using the Q-Learning networks as defined above in the sections `DQLNet` and `DQLSimple` the state and the actions need to be converted into the input format accepted by the Q-Learning network.\n",
    "The function below merges both input values and gets them into the proper format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 326,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "37.2 ms ± 1.67 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n",
      "peak memory: 1014.62 MiB, increment: 0.23 MiB\n"
     ]
    }
   ],
   "source": [
    "def action_to_q_learning_format(\n",
    "    board_history: np.ndarray, action_history: np.ndarray\n",
    ") -> np.ndarray:\n",
    "    \"\"\"Formats the board history and the action history into Q-Learning inputs.\n",
    "\n",
    "    Args:\n",
    "        board_history: A stack of board histories.\n",
    "        action_history: A stack of action histories.\n",
    "    \"\"\"\n",
    "    q_learning_format = np.zeros(\n",
    "        (SIMULATE_TURNS, board_history.shape[1], 2, 8, 8), dtype=float\n",
    "    )\n",
    "    q_learning_format[:, :, 0, :, :] = board_history\n",
    "    q_learning_format[:, :, 1, :, :] = -1\n",
    "\n",
    "    game_index = list(range(board_history.shape[1]))\n",
    "    for turn_index in range(SIMULATE_TURNS):\n",
    "        q_learning_format[\n",
    "            turn_index,\n",
    "            game_index,\n",
    "            1,\n",
    "            action_history[turn_index, game_index, 0],\n",
    "            action_history[turn_index, game_index, 1],\n",
    "        ] = 1\n",
    "    return q_learning_format\n",
    "\n",
    "\n",
    "%timeit action_to_q_learning_format(_board_history, _action_history)\n",
    "%memit action_to_q_learning_format(_board_history, _action_history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploiting the game symetry for data generation\n",
    "The game's axis ambivalence allows for the generation of up to $2^3$ unique boards by using symmetries. The symmetry axes are vertical and horizontal flips, as well as the transposed versions. This results in $2^3$ symmetry variations that can be used to train the artificial neural network (ANN). Instead of using all $2^3$ variations, a random selection is taken to break the strict sequence and facilitate training. This functions also converts into the proper format for Q-Learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "61.5 ms ± 4.81 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n",
      "peak memory: 441.80 MiB, increment: 0.36 MiB\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(2, 2, 2, 70, 100, 2, 8, 8)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def build_symetry_action(\n",
    "    board_history: np.ndarray, action_history: np.ndarray\n",
    ") -> np.ndarray:\n",
    "    \"\"\"Build a set of symmetrical game histories in the style of a q_learning learning policy.\n",
    "\n",
    "    Args:\n",
    "        board_history: A history of game states.\n",
    "        action_history: A history fo action states.\n",
    "    \"\"\"\n",
    "    board_history = board_history.copy()\n",
    "    board_history[1::2] *= -1\n",
    "    q_learning_format = np.zeros(\n",
    "        (2, 2, 2, SIMULATE_TURNS, board_history.shape[1], 2, 8, 8)\n",
    "    )\n",
    "    q_learning_format[0, 0, 0, :, :, :, :, :] = action_to_q_learning_format(\n",
    "        board_history, action_history\n",
    "    )\n",
    "    q_learning_format[1, 0, 0, :, :, :, :, :] = np.transpose(\n",
    "        q_learning_format[0, 0, 0, :, :, :, :, :], [0, 1, 2, 4, 3]\n",
    "    )\n",
    "    q_learning_format[:, 1, 0, :, :, :, :, :] = q_learning_format[\n",
    "        :, 0, 0, :, :, :, ::-1, :\n",
    "    ]\n",
    "    q_learning_format[:, :, 1, :, :, :, :, :] = q_learning_format[\n",
    "        :, :, 0, :, :, :, :, ::-1\n",
    "    ]\n",
    "    return q_learning_format\n",
    "\n",
    "\n",
    "%timeit build_symetry_action(_board_history, _action_history)\n",
    "%memit build_symetry_action(_board_history, _action_history)\n",
    "build_symetry_action(_board_history, _action_history).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During the training of an ANN, it is necessary to monitor its progress by tracking its loss. Additional metrics are also used to measure significant results separately, since we work with a composite reward in reinforcement learning. Therefore, the final score and win ratio are tracked and can be plotted using the function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-03-31T00:14:24.420801Z",
     "start_time": "2023-03-31T00:14:24.370778Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def live_history(training_history: pd.DataFrame, ai_name: str | None = None):\n",
    "    \"\"\"Plots the training history of a Q-Learning Policy.\n",
    "\n",
    "    Args:\n",
    "        training_history: The training history of a Q-Learning Policy.\n",
    "        ai_name: The optional name of the policy.\n",
    "    \"\"\"\n",
    "    training_history.columns = training_history.columns.to_series().apply(\n",
    "        lambda x: \"{0} {1}\".format(*x)\n",
    "    )\n",
    "    clear_output(wait=True)\n",
    "    fig, (ax1, ax2, ax3) = plt.subplots(ncols=3, figsize=(14, 7))\n",
    "    _ = (\n",
    "        training_history[\n",
    "            [c for c in training_history.columns if c.endswith(\"final_score\")]\n",
    "        ]\n",
    "        .rename(lambda c: c.split(\" \")[0], axis=1)\n",
    "        .plot(ax=ax1)\n",
    "    )\n",
    "    ax1.set_title(\"Final score\")\n",
    "    ax1.set_xlabel(\"epochs\")\n",
    "    ax1.set_ylabel(\"Average final score\")\n",
    "    _ = (\n",
    "        training_history[[c for c in training_history.columns if c.endswith(\"win\")]]\n",
    "        .rename(lambda c: c.split(\" \")[0], axis=1)\n",
    "        .plot(ax=ax2)\n",
    "    )\n",
    "    ax2.set_title(\"Win score\")\n",
    "    ax2.set_xlabel(\"epochs\")\n",
    "    ax2.set_ylabel(\"Average signum winner\")\n",
    "    _ = (\n",
    "        np.sqrt(\n",
    "            training_history[\n",
    "                [c for c in training_history.columns if c.startswith(\"loss\")]\n",
    "            ]\n",
    "        )\n",
    "        .rename(lambda c: c.split(\" \")[1], axis=1)\n",
    "        .plot(ax=ax3)\n",
    "    )\n",
    "    ax3.set_yscale(\"log\")\n",
    "    ax3.set_xlabel(\"epochs\")\n",
    "    ax3.set_ylabel(\"MSE\")\n",
    "    ax3.set_title(\"Loss\")\n",
    "    fig.suptitle(f\"Training history of {ai_name}\" if ai_name else f\"Training history\")\n",
    "    fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The QLPolicy Class\n",
    "\n",
    "With the elements created above, we can now assemble a cohesive class that encompasses all the functionality needed to train a simple Q-Learning model. The class includes functions for training as a whole, per epoch, and per batch, as well as tools for analysis, internal policies, and logging the training history and quality monitoring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "class QLPolicy(GamePolicy):\n",
    "    \"\"\"\n",
    "    A simple Q-Learning policy.\n",
    "    \"\"\"\n",
    "\n",
    "    # noinspection PyProtectedMember\n",
    "    def __init__(\n",
    "        self,\n",
    "        epsilon: float,\n",
    "        neural_network: DQLNet | DQLSimple,\n",
    "        symmetry_mode: SymmetryMode,\n",
    "        gamma: float = 0.8,\n",
    "        who_won_fraction: float = 0,\n",
    "        final_score_fraction: float = 0,\n",
    "        optimizer: torch.optim.Optimizer | None = None,\n",
    "        loss: nn.modules.loss._Loss | None = None,\n",
    "    ):\n",
    "        \"\"\"Creates a Q-learning policy using the specified arguments.\n",
    "\n",
    "        Args:\n",
    "            epsilon: Defines how greedy the policy should act. A value of 1 means the policy is completely greedy.\n",
    "            neural_network: The neural network structure used for this policy.\n",
    "            symmetry_mode: The symmetry mode used to select data. Choose from 'all_symmetric_variances' or 'random_choice'.\n",
    "            gamma: The discount factor used in the Q-learning algorithm.\n",
    "            who_won_fraction: The weight of the reward based on who won the game.\n",
    "            final_score_fraction: The weight of the reward based on the final score of the game.\n",
    "            optimizer: The optimizer used to train the model.\n",
    "            loss: The loss function used to train the model. None defaults to MSE.\n",
    "        \"\"\"\n",
    "        super().__init__(epsilon)\n",
    "        assert 0 <= gamma <= 1\n",
    "        self.gamma: float = gamma\n",
    "        del gamma\n",
    "        self.symmetry_mode: SymmetryMode = symmetry_mode\n",
    "        del symmetry_mode\n",
    "        self.neural_network: DQLNet | DQLSimple = neural_network\n",
    "        del neural_network\n",
    "        self.who_won_fraction: float = who_won_fraction\n",
    "        del who_won_fraction\n",
    "        self.final_score_fraction: float = final_score_fraction\n",
    "        del final_score_fraction\n",
    "\n",
    "        if optimizer is None:\n",
    "            self.optimizer = torch.optim.Adam(self.neural_network.parameters(), lr=5e-5)\n",
    "        else:\n",
    "            self.optimizer = optimizer\n",
    "        if loss is None:\n",
    "            self.loss = nn.MSELoss()\n",
    "        else:\n",
    "            self.loss = loss\n",
    "        self.training_results: list[dict[tuple[str, str], float]] = []\n",
    "        self.loss_history: list[float] = []\n",
    "\n",
    "    @property\n",
    "    def policy_name(self) -> str:\n",
    "        \"\"\"Generates a name for a Q-Learning policy from all their arguments\"\"\"\n",
    "        symmetry_name = {SymmetryMode.MULTIPLY: \"M\", SymmetryMode.BREAK_SEQUENCE: \"B\"}\n",
    "        g = f\"{self.gamma:.1f}\".replace(\".\", \"\")\n",
    "        ww = f\"{self.who_won_fraction:.1f}\".replace(\".\", \"\")\n",
    "        fsf = f\"{self.final_score_fraction:.1f}\".replace(\".\", \"\")\n",
    "        return f\"QL-{symmetry_name[self.symmetry_mode]}-G{g}-WW{ww}-FSF{fsf}-{self.neural_network.__class__.__name__}-{self.loss.__class__.__name__}\"\n",
    "\n",
    "    def __repr__(self) -> str:\n",
    "        return self.policy_name\n",
    "\n",
    "    def __str__(self) -> str:\n",
    "        return self.policy_name\n",
    "\n",
    "    def _internal_policy(self, boards: np.ndarray) -> np.ndarray:\n",
    "        \"\"\"Calculates the q-learning values for all the possible actions.\n",
    "\n",
    "        Args:\n",
    "            boards: A stack of game boards.\n",
    "        \"\"\"\n",
    "        results = np.zeros_like(boards, dtype=float)\n",
    "        results = torch.from_numpy(results).float()\n",
    "        q_learning_boards = np.zeros((boards.shape[0], 2, 8, 8))\n",
    "        q_learning_boards[:, 0, :, :] = boards\n",
    "        poss_turns = boards == 0  # checks where fields are empty.\n",
    "        poss_turns &= binary_dilation(boards == -1, SURROUNDING)\n",
    "        turn_possible = np.any(poss_turns, axis=0)\n",
    "        for action_x, action_y in itertools.product(range(8), range(8)):\n",
    "            if not turn_possible[action_x, action_y]:\n",
    "                continue\n",
    "            _q_learning_board = q_learning_boards[\n",
    "                poss_turns[range(boards.shape[0]), action_x, action_y]\n",
    "            ].copy()\n",
    "            _q_learning_board[\n",
    "                range(_q_learning_board.shape[0]), 1, action_x, action_y\n",
    "            ] = 1\n",
    "\n",
    "            ql_result = self.neural_network.forward(_q_learning_board)\n",
    "            results[poss_turns[:, action_x, action_y], action_x, action_y] = (\n",
    "                ql_result.reshape(-1) + 0.1\n",
    "            )\n",
    "        return results.cpu().detach().numpy()\n",
    "\n",
    "    def generate_trainings_data(\n",
    "        self, generate_data_size: int\n",
    "    ) -> tuple[torch.Tensor, torch.Tensor]:\n",
    "        \"\"\"Generates trainings data.\n",
    "\n",
    "        Args:\n",
    "            generate_data_size: The number of games that should be used to generate trainingsdata.\n",
    "        \"\"\"\n",
    "        train_boards, train_actions = simulate_game(generate_data_size, (self, self))\n",
    "        action_possible = ~np.all(train_actions[:, :] == -1, axis=2)\n",
    "        q_leaning_formatted_action = build_symetry_action(train_boards, train_actions)\n",
    "        q_rewords = calculate_q_reword(\n",
    "            board_history=train_boards,\n",
    "            who_won_fraction=self.who_won_fraction,\n",
    "            final_score_fraction=self.final_score_fraction,\n",
    "        )\n",
    "        q_rewords[1::2, :] *= -1\n",
    "        if self.symmetry_mode == SymmetryMode.MULTIPLY:\n",
    "            new_q_rewords = np.zeros((2, 2, 2) + q_rewords.shape)\n",
    "            for i, k, j in itertools.product((0, 1), (0, 1), (0, 1)):\n",
    "                new_q_rewords[i, k, j] = q_rewords\n",
    "            q_rewords = new_q_rewords\n",
    "            action_possible = np.array([action_possible] * 8).reshape(-1)\n",
    "\n",
    "        elif self.symmetry_mode == SymmetryMode.BREAK_SEQUENCE:\n",
    "            axis1 = np.random.randint(0, high=2, size=SIMULATE_TURNS, dtype=int)\n",
    "            axis2 = np.random.randint(0, high=2, size=SIMULATE_TURNS, dtype=int)\n",
    "            axis3 = np.random.randint(0, high=2, size=SIMULATE_TURNS, dtype=int)\n",
    "            q_leaning_formatted_action = q_leaning_formatted_action[\n",
    "                axis1, axis2, axis3, range(SIMULATE_TURNS)\n",
    "            ]\n",
    "            action_possible = action_possible.reshape(-1)\n",
    "\n",
    "        return (\n",
    "            torch.from_numpy(\n",
    "                q_leaning_formatted_action.reshape(-1, 2, BOARD_SIZE, BOARD_SIZE)[\n",
    "                    action_possible\n",
    "                ]\n",
    "            ).float(),\n",
    "            torch.from_numpy(q_rewords.reshape(-1, 1)[action_possible]).float(),\n",
    "        )\n",
    "\n",
    "    def train_batch(self, nr_of_games: int) -> None:\n",
    "        \"\"\"Trains a single batch of on a set of games.\n",
    "\n",
    "        Args:\n",
    "            nr_of_games: The number of games that should be played to train the model.\n",
    "        \"\"\"\n",
    "        x_train, y_train = self.generate_trainings_data(nr_of_games)\n",
    "        y_pred = self.neural_network.forward(x_train)\n",
    "        loss_score = self.loss(y_pred, y_train)\n",
    "        self.optimizer.zero_grad()\n",
    "        self.loss_history.append(loss_score.item())\n",
    "        loss_score.backward()\n",
    "        # Update the parameters\n",
    "        self.optimizer.step()\n",
    "        # generate trainings data\n",
    "\n",
    "    def evaluate_model(self, compare_policies: list[GamePolicy], nr_of_games: int):\n",
    "        \"\"\"Evaluates a model against a list of policies.\n",
    "\n",
    "        Args:\n",
    "            compare_policies:\n",
    "            nr_of_games:\n",
    "        \"\"\"\n",
    "        result_dict: dict[tuple[str, str], float] = {}\n",
    "        eval_copy = copy.copy(self)\n",
    "        eval_copy.epsilon = 1\n",
    "        for model in compare_models:\n",
    "            boards_black, _ = simulate_game(nr_of_games, (eval_copy, model))\n",
    "            boards_white, _ = simulate_game(nr_of_games, (model, eval_copy))\n",
    "            win_eval_white = calculate_who_won(boards_white)\n",
    "            win_eval_black = calculate_who_won(boards_black)\n",
    "            result_dict[(model.policy_name, \"final_score\")] = (\n",
    "                float(\n",
    "                    np.mean(\n",
    "                        calculate_final_evaluation_for_history(boards_black)\n",
    "                        + (calculate_final_evaluation_for_history(boards_white) * -1)\n",
    "                    )\n",
    "                )\n",
    "                * 64\n",
    "            )\n",
    "            result_dict[(model.policy_name, \"white_win\")] = (\n",
    "                np.sum(win_eval_white == -1) / nr_of_games\n",
    "            )\n",
    "            result_dict[(model.policy_name, \"white_lose\")] = (\n",
    "                np.sum(win_eval_white == 1) / nr_of_games\n",
    "            )\n",
    "            result_dict[(model.policy_name, \"black_win\")] = (\n",
    "                np.sum(win_eval_black == 1) / nr_of_games\n",
    "            )\n",
    "            result_dict[(model.policy_name, \"black_lose\")] = (\n",
    "                np.sum(win_eval_black == -1) / nr_of_games\n",
    "            )\n",
    "\n",
    "        result_dict[(\"loss\", \"mean\")] = float(np.mean(np.array(self.loss_history)))\n",
    "        result_dict[(\"loss\", \"min\")] = np.min(np.array(self.loss_history))\n",
    "        result_dict[(\"loss\", \"max\")] = np.max(np.array(self.loss_history))\n",
    "        self.loss_history = []\n",
    "        result_dict[(\"base\", \"base\")] = nr_of_games\n",
    "        return result_dict\n",
    "\n",
    "    def save(self):\n",
    "        \"\"\"Saves the current model.\"\"\"\n",
    "        filename: str = f\"{self.policy_name}-{len(self.training_results)}\"\n",
    "        with open(TRAINING_RESULT_PATH / Path(f\"{filename}.pickle\"), \"wb\") as f:\n",
    "            pickle.dump(self.training_results, f)\n",
    "        torch.save(\n",
    "            self.neural_network.state_dict(),\n",
    "            TRAINING_RESULT_PATH / Path(f\"{filename}.torch\"),\n",
    "        )\n",
    "\n",
    "    def load(self):\n",
    "        \"\"\"Loads the latest iteration of a model with the same configuration\"\"\"\n",
    "        pickle_files = glob.glob(f\"{TRAINING_RESULT_PATH}/{self.policy_name}-*.pickle\")\n",
    "        torch_files = glob.glob(f\"{TRAINING_RESULT_PATH}/{self.policy_name}-*.torch\")\n",
    "\n",
    "        assert len(pickle_files) == len(torch_files)\n",
    "        if not pickle_files:\n",
    "            return\n",
    "\n",
    "        pickle_dict = {\n",
    "            int(file.split(\"-\")[-1].split(\".\")[0]): file for file in pickle_files\n",
    "        }\n",
    "        torch_dict = {\n",
    "            int(file.split(\"-\")[-1].split(\".\")[0]): file for file in torch_files\n",
    "        }\n",
    "        pickle_file = pickle_dict[max(pickle_dict.keys())]\n",
    "        torch_file = torch_dict[max(torch_dict.keys())]\n",
    "\n",
    "        with open(pickle_file, \"rb\") as f:\n",
    "            self.training_results = pickle.load(f)\n",
    "\n",
    "        self.neural_network.load_state_dict(torch.load(Path(torch_file)))\n",
    "\n",
    "    def train(\n",
    "        self,\n",
    "        epochs: int,\n",
    "        batches: int,\n",
    "        batch_size: int,\n",
    "        eval_batch_size: int,\n",
    "        compare_with: list[GamePolicy],\n",
    "        save_every_epoch: bool = True,\n",
    "        live_plot: bool = True,\n",
    "    ) -> pd.DataFrame:\n",
    "        \"\"\"Trains a model.\n",
    "\n",
    "        Args:\n",
    "            epochs: The number of epochs to train the model.\n",
    "            batches: The number of batches per epoch.\n",
    "            batch_size: The number of games per epoch.\n",
    "            eval_batch_size: The number of samples that should be made\n",
    "            compare_with: A list of static policies that should be used to evaluate training results.\n",
    "            save_every_epoch: If the policy should be saved every epoch.\n",
    "            live_plot: If policy should be plotted every epoch.\n",
    "        \"\"\"\n",
    "        assert epochs > 0\n",
    "        assert eval_batch_size >= 0\n",
    "        epoch_progress = tqdm(range(epochs), unit=\"epoch\")\n",
    "        for _ in epoch_progress:\n",
    "            for _ in tqdm(range(batches), unit=\"batch\"):\n",
    "                self.train_batch(batch_size)\n",
    "            self.training_results.append(\n",
    "                self.evaluate_model(compare_with, eval_batch_size)\n",
    "            )\n",
    "            if save_every_epoch:\n",
    "                self.save()\n",
    "            if live_plot:\n",
    "                self.plot_history()\n",
    "                plt.show()\n",
    "            display(epoch_progress.container)\n",
    "        return self.history\n",
    "\n",
    "    def plot_history(self) -> None:\n",
    "        \"\"\"Plots the training history.\"\"\"\n",
    "        if not self.training_results:\n",
    "            return None\n",
    "        return live_history(self.history, str(self))\n",
    "\n",
    "    @property\n",
    "    def history(self) -> pd.DataFrame:\n",
    "        \"\"\"Generates the history from data\"\"\"\n",
    "        if not self.training_results:\n",
    "            return pd.DataFrame()\n",
    "        pandas_result = pd.DataFrame(self.training_results)\n",
    "        pandas_result.columns = pd.MultiIndex.from_tuples(pandas_result.columns)\n",
    "        return pandas_result\n",
    "\n",
    "\n",
    "ql_policy1 = QLPolicy(\n",
    "    0.95,\n",
    "    neural_network=DQLNet(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=0.8,\n",
    "    who_won_fraction=1,\n",
    "    final_score_fraction=0,\n",
    ")\n",
    "\n",
    "assert copy.copy(ql_policy1) is not ql_policy1\n",
    "assert copy.copy(ql_policy1).neural_network is ql_policy1.neural_network\n",
    "\n",
    "# noinspection PyProtectedMember\n",
    "t1, t2 = ql_policy1._internal_policy(get_new_games(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calibrating the / greedy factor ($\\epsilon$)\n",
    "\n",
    "Given the policy of not using memory replay, it is advisable to set the greedy factor $\\epsilon$ high enough such that some policies remain unchanged until the end of the simulation. Assuming a simulation of the game with 1000 probes per batch, trial and error suggests a minimum greedy factor of 0.92.\n",
    "\n",
    "However, further experimentation revealed significant instability in the ANN models and poor convergence, leading to low win ratios and final scores. To address this issue, increasing the number of samples per batch or implementing a memory replay buffer may be beneficial. More on this will be discussed later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 29628 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "probes: int = 1000\n",
    "_ = (\n",
    "    calculate_board_branching(simulate_game(probes, (ql_policy1, ql_policy1))[0])\n",
    "    / probes\n",
    ").plot(\n",
    "    ylim=(0, 1),\n",
    "    xlim=(0, 69),\n",
    "    title=f\"Branches for a QL policy with epsilon={ql_policy1.epsilon}\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training some polices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining some polices\n",
    "In varius stages of this project varius Policies where created. Those compose of different values for exploration, reword fractions and gamma values and different network structures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QL-M-G09-WW00-FSF10-DQLSimple-MSELoss'"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ql_policy1 = QLPolicy(\n",
    "    0.92,\n",
    "    neural_network=DQLSimple(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=0.9,\n",
    "    who_won_fraction=0,\n",
    "    final_score_fraction=1,\n",
    ")\n",
    "ql_policy1.policy_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ql_policies = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QL-M-G09-WW00-FSF10-DQLSimple-MSELoss'"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ql_policy1 = QLPolicy(\n",
    "    0.92,\n",
    "    neural_network=DQLSimple(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=0.9,\n",
    "    who_won_fraction=0,\n",
    "    final_score_fraction=1,\n",
    ")\n",
    "ql_policies.append(ql_policy1)\n",
    "ql_policy1.policy_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QL-M-G08-WW00-FSF10-DQLSimple-MSELoss'"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ql_policy2 = QLPolicy(\n",
    "    0.92,\n",
    "    neural_network=DQLSimple(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=0.8,\n",
    "    who_won_fraction=0,\n",
    "    final_score_fraction=1,\n",
    ")\n",
    "ql_policies.append(ql_policy2)\n",
    "ql_policy2.policy_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QL-M-G10-WW00-FSF10-DQLSimple-MSELoss'"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ql_policy3 = QLPolicy(\n",
    "    0.92,\n",
    "    neural_network=DQLSimple(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=1,\n",
    "    who_won_fraction=0,\n",
    "    final_score_fraction=1,\n",
    ")\n",
    "ql_policies.append(ql_policy3)\n",
    "ql_policy3.policy_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QL-M-G09-WW10-FSF00-DQLSimple-MSELoss'"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ql_policy4 = QLPolicy(\n",
    "    0.92,\n",
    "    neural_network=DQLSimple(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=0.9,\n",
    "    who_won_fraction=1,\n",
    "    final_score_fraction=0,\n",
    ")\n",
    "ql_policies.append(ql_policy4)\n",
    "ql_policy4.policy_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QL-M-G09-WW03-FSF03-DQLSimple-MSELoss'"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ql_policy5 = QLPolicy(\n",
    "    0.95,\n",
    "    neural_network=DQLSimple(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=0.9,\n",
    "    who_won_fraction=0.3,\n",
    "    final_score_fraction=0.3,\n",
    ")\n",
    "ql_policies.append(ql_policy5)\n",
    "ql_policy5.policy_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QL-M-G09-WW03-FSF07-DQLSimple-MSELoss'"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ql_policy6 = QLPolicy(\n",
    "    0.95,\n",
    "    neural_network=DQLSimple(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=0.92,\n",
    "    who_won_fraction=0.3,\n",
    "    final_score_fraction=0.65,\n",
    ")\n",
    "ql_policies.append(ql_policy6)\n",
    "ql_policy6.policy_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QL-M-G09-WW02-FSF07-DQLSimple-MSELoss'"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ql_policy7 = QLPolicy(\n",
    "    0.95,\n",
    "    neural_network=DQLSimple(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=0.92,\n",
    "    who_won_fraction=0.2,\n",
    "    final_score_fraction=0.65,\n",
    ")\n",
    "ql_policies.append(ql_policy7)\n",
    "ql_policy7.policy_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QL-M-G08-WW00-FSF10-DQLNet-MSELoss'"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ql_policy8 = QLPolicy(\n",
    "    0.92,\n",
    "    neural_network=DQLNet(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=0.8,\n",
    "    who_won_fraction=0,\n",
    "    final_score_fraction=1,\n",
    ")\n",
    "ql_policies.append(ql_policy8)\n",
    "ql_policy8.policy_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QL-M-G10-WW00-FSF10-DQLNet-MSELoss'"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ql_policy9 = QLPolicy(\n",
    "    0.92,\n",
    "    neural_network=DQLNet(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=1,\n",
    "    who_won_fraction=0,\n",
    "    final_score_fraction=1,\n",
    ")\n",
    "ql_policies.append(ql_policy9)\n",
    "ql_policy9.policy_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QL-M-G09-WW10-FSF00-DQLNet-MSELoss'"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ql_policy10 = QLPolicy(\n",
    "    0.92,\n",
    "    neural_network=DQLNet(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=0.9,\n",
    "    who_won_fraction=1,\n",
    "    final_score_fraction=0,\n",
    ")\n",
    "ql_policies.append(ql_policy10)\n",
    "ql_policy10.policy_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QL-M-G09-WW03-FSF03-DQLNet-MSELoss'"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ql_policy11 = QLPolicy(\n",
    "    0.95,\n",
    "    neural_network=DQLNet(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=0.9,\n",
    "    who_won_fraction=0.3,\n",
    "    final_score_fraction=0.3,\n",
    ")\n",
    "ql_policies.append(ql_policy11)\n",
    "ql_policy11.policy_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QL-M-G09-WW03-FSF07-DQLNet-MSELoss'"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ql_policy12 = QLPolicy(\n",
    "    0.95,\n",
    "    neural_network=DQLNet(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=0.92,\n",
    "    who_won_fraction=0.3,\n",
    "    final_score_fraction=0.65,\n",
    ")\n",
    "ql_policies.append(ql_policy12)\n",
    "ql_policy12.policy_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'QL-M-G09-WW02-FSF07-DQLNet-MSELoss'"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ql_policy13 = QLPolicy(\n",
    "    0.95,\n",
    "    neural_network=DQLNet(),\n",
    "    symmetry_mode=SymmetryMode.MULTIPLY,\n",
    "    gamma=0.92,\n",
    "    who_won_fraction=0.2,\n",
    "    final_score_fraction=0.65,\n",
    ")\n",
    "ql_policies.append(ql_policy13)\n",
    "ql_policy13.policy_name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining some metric polices\n",
    "\n",
    "Since the ANN policies are in flux and change while trainig a continues comparison with unchanging policies is needed. The `RandomPolicy` and the `GreedyPolicy` fulfill that role."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "constant_metric_policies = [RandomPolicy(0), GreedyPolicy(0)]"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "for i in range(100):\n",
    "    for ql_policy in ql_policys:\n",
    "        ql_policy.load()\n",
    "        ql_policy.train(1, 10, 1000, 250, constant_metric_policies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below the training progress for the Q-Learning policy is shown."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 263016 characters]\n",
      "text/plain": [
       "<Figure size 1400x700 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 231500 characters]\n",
      "text/plain": [
       "<Figure size 1400x700 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 201244 characters]\n",
      "text/plain": [
       "<Figure size 1400x700 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 194584 characters]\n",
      "text/plain": [
       "<Figure size 1400x700 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 216188 characters]\n",
      "text/plain": [
       "<Figure size 1400x700 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 207264 characters]\n",
      "text/plain": [
       "<Figure size 1400x700 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 184040 characters]\n",
      "text/plain": [
       "<Figure size 1400x700 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 203068 characters]\n",
      "text/plain": [
       "<Figure size 1400x700 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 179600 characters]\n",
      "text/plain": [
       "<Figure size 1400x700 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 204036 characters]\n",
      "text/plain": [
       "<Figure size 1400x700 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 195504 characters]\n",
      "text/plain": [
       "<Figure size 1400x700 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 220220 characters]\n",
      "text/plain": [
       "<Figure size 1400x700 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 191400 characters]\n",
      "text/plain": [
       "<Figure size 1400x700 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for ql_policy in ql_policies:\n",
    "    ql_policy.load()\n",
    "    ql_policy.plot_history()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training consistency varies from policy to policy, with some policies being more stable and exhibiting less variation. Moreover, the highly interdependent metrics indicate that the evaluation sample size may be too small.\n",
    "\n",
    "Since batch size and metrics have similar sizes this suggests that the batch size should also be increased. Sadly with the current Q-Learning setup this is not possible due to CPU constraints."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example simulations\n",
    "\n",
    "The section below shows the trained policies, playing as black against a white opponent.\n",
    "The Policy can be chosen via the dropdown menu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2b28e396130d40c88f8b8f67adb86962",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(Dropdown(description='ql_policy', options=(QL-M-G09-WW00-FSF10-DQLSimple-MSELoss, QL-M-G…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "@interact(ql_policy=ql_policies)\n",
    "def plot_policy_example(ql_policy):\n",
    "    ql_policy.load()\n",
    "    np.random.seed(1)\n",
    "    ql_policy = copy.copy(ql_policy)\n",
    "    ql_policy.epsilon = 1\n",
    "    simulation_results = simulate_game(1, (ql_policy, RandomPolicy(0)))\n",
    "    _unique_bords, _unique_actions = drop_duplicate_boards(\n",
    "        simulation_results[0].reshape(-1, 8, 8), simulation_results[1].reshape(-1, 2)\n",
    "    )\n",
    "    plot_othello_boards(_unique_bords, actions=_unique_actions)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For some analysis we need a specially good policy. Normally "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss"
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_policy = ql_policies[4]\n",
    "example_policy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis of the Policies\n",
    "\n",
    "When creating such policies it is necessary to evaluate the results. One way of doing so is to play a huge tournament where all Policies cross paths at least twice.\n",
    "The resulting dataframe contains the results as a json where on the index the first policy is noted and the second by the column.\n",
    "This allows for different metrics for all the policies while simulating them only once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f7a426f9cd7448e9813b0d6b84b4465d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/225 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "policy_list = constant_metric_policies + ql_policies\n",
    "\n",
    "RESULTS_FILE: Final[str] = \"results.pickle\"\n",
    "if not os.path.exists(RESULTS_FILE):\n",
    "    result_df = pd.DataFrame(\n",
    "        index=[policy.policy_name for policy in policy_list],\n",
    "        columns=[policy.policy_name for policy in policy_list],\n",
    "    )\n",
    "else:\n",
    "    result_df = pd.read_pickle(RESULTS_FILE)\n",
    "\n",
    "nr_of_eval_games = 250\n",
    "\n",
    "for policy1, policy2 in tqdm(list(itertools.product(policy_list, policy_list))):\n",
    "    if not pd.isna(result_df.at[policy1.policy_name, policy2.policy_name]):\n",
    "        continue\n",
    "    _result_dict = {}\n",
    "    policy1 = copy.copy(policy1)\n",
    "    policy1.epsilon = 0.99\n",
    "    policy2 = copy.copy(policy2)\n",
    "    policy2.epsilon = 0.99\n",
    "    _boards_black, _ = simulate_game(nr_of_eval_games, (policy1, policy2))\n",
    "    _win_eval_black = calculate_who_won(_boards_black)\n",
    "    _result_dict[\"final_score\"] = float(\n",
    "        np.mean(calculate_final_evaluation_for_history(_boards_black))\n",
    "    )\n",
    "    _result_dict[\"win\"] = np.sum(_win_eval_black == 1) / nr_of_eval_games\n",
    "    _result_dict[\"lose\"] = np.sum(_win_eval_black == -1) / nr_of_eval_games\n",
    "    result_df.at[policy1.policy_name, policy2.policy_name] = _result_dict\n",
    "    result_df.to_pickle(RESULTS_FILE)\n",
    "result_df.index.name = \"Black policy\"\n",
    "result_df.columns.name = \"White policy\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Analysing the policy by the win score\n",
    "The table presented below displays the win ratio for each color, providing directional insight into how one policy played against the other as black or white. While most values are positive, indicating that the player who played as black had an advantage, this bias could not be overcome with the current architecture. Nonetheless, the table does demonstrate some learning behavior."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>White policy</th>\n",
       "      <th>random</th>\n",
       "      <th>greedy_policy</th>\n",
       "      <th>QL-M-G09-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G08-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G10-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G09-WW10-FSF00-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G09-WW03-FSF03-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G09-WW03-FSF07-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G09-WW02-FSF07-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G08-WW00-FSF10-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G10-WW00-FSF10-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G09-WW10-FSF00-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G09-WW03-FSF03-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G09-WW03-FSF07-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G09-WW02-FSF07-DQLNet-MSELoss</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Black policy</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>random</th>\n",
       "      <td>0.432</td>\n",
       "      <td>0.376</td>\n",
       "      <td>0.340</td>\n",
       "      <td>0.312</td>\n",
       "      <td>0.368</td>\n",
       "      <td>0.344</td>\n",
       "      <td>0.244</td>\n",
       "      <td>0.332</td>\n",
       "      <td>0.312</td>\n",
       "      <td>0.452</td>\n",
       "      <td>0.372</td>\n",
       "      <td>0.272</td>\n",
       "      <td>0.352</td>\n",
       "      <td>0.388</td>\n",
       "      <td>0.356</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>greedy_policy</th>\n",
       "      <td>0.592</td>\n",
       "      <td>0.488</td>\n",
       "      <td>0.596</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.452</td>\n",
       "      <td>0.544</td>\n",
       "      <td>0.384</td>\n",
       "      <td>0.436</td>\n",
       "      <td>0.484</td>\n",
       "      <td>0.536</td>\n",
       "      <td>0.472</td>\n",
       "      <td>0.432</td>\n",
       "      <td>0.580</td>\n",
       "      <td>0.508</td>\n",
       "      <td>0.444</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <td>0.592</td>\n",
       "      <td>0.456</td>\n",
       "      <td>0.856</td>\n",
       "      <td>0.172</td>\n",
       "      <td>0.748</td>\n",
       "      <td>0.076</td>\n",
       "      <td>0.136</td>\n",
       "      <td>0.760</td>\n",
       "      <td>0.176</td>\n",
       "      <td>0.136</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.096</td>\n",
       "      <td>0.804</td>\n",
       "      <td>0.908</td>\n",
       "      <td>0.720</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G08-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <td>0.616</td>\n",
       "      <td>0.384</td>\n",
       "      <td>0.852</td>\n",
       "      <td>0.792</td>\n",
       "      <td>0.832</td>\n",
       "      <td>0.160</td>\n",
       "      <td>0.812</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.088</td>\n",
       "      <td>0.224</td>\n",
       "      <td>0.744</td>\n",
       "      <td>0.076</td>\n",
       "      <td>0.828</td>\n",
       "      <td>0.756</td>\n",
       "      <td>0.108</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G10-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <td>0.580</td>\n",
       "      <td>0.532</td>\n",
       "      <td>0.796</td>\n",
       "      <td>0.772</td>\n",
       "      <td>0.804</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.172</td>\n",
       "      <td>0.108</td>\n",
       "      <td>0.888</td>\n",
       "      <td>0.840</td>\n",
       "      <td>0.700</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.720</td>\n",
       "      <td>0.188</td>\n",
       "      <td>0.744</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW10-FSF00-DQLSimple-MSELoss</th>\n",
       "      <td>0.520</td>\n",
       "      <td>0.340</td>\n",
       "      <td>0.820</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.180</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.204</td>\n",
       "      <td>0.780</td>\n",
       "      <td>0.716</td>\n",
       "      <td>0.876</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.088</td>\n",
       "      <td>0.800</td>\n",
       "      <td>0.176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW03-FSF03-DQLSimple-MSELoss</th>\n",
       "      <td>0.628</td>\n",
       "      <td>0.616</td>\n",
       "      <td>0.756</td>\n",
       "      <td>0.124</td>\n",
       "      <td>0.204</td>\n",
       "      <td>0.220</td>\n",
       "      <td>0.204</td>\n",
       "      <td>0.224</td>\n",
       "      <td>0.872</td>\n",
       "      <td>0.932</td>\n",
       "      <td>0.824</td>\n",
       "      <td>0.136</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.236</td>\n",
       "      <td>0.208</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW03-FSF07-DQLSimple-MSELoss</th>\n",
       "      <td>0.576</td>\n",
       "      <td>0.528</td>\n",
       "      <td>0.144</td>\n",
       "      <td>0.800</td>\n",
       "      <td>0.184</td>\n",
       "      <td>0.108</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.256</td>\n",
       "      <td>0.240</td>\n",
       "      <td>0.164</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.108</td>\n",
       "      <td>0.096</td>\n",
       "      <td>0.180</td>\n",
       "      <td>0.756</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW02-FSF07-DQLSimple-MSELoss</th>\n",
       "      <td>0.524</td>\n",
       "      <td>0.404</td>\n",
       "      <td>0.876</td>\n",
       "      <td>0.704</td>\n",
       "      <td>0.088</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.064</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.128</td>\n",
       "      <td>0.916</td>\n",
       "      <td>0.192</td>\n",
       "      <td>0.776</td>\n",
       "      <td>0.220</td>\n",
       "      <td>0.720</td>\n",
       "      <td>0.908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G08-WW00-FSF10-DQLNet-MSELoss</th>\n",
       "      <td>0.492</td>\n",
       "      <td>0.452</td>\n",
       "      <td>0.112</td>\n",
       "      <td>0.128</td>\n",
       "      <td>0.072</td>\n",
       "      <td>0.844</td>\n",
       "      <td>0.052</td>\n",
       "      <td>0.088</td>\n",
       "      <td>0.948</td>\n",
       "      <td>0.196</td>\n",
       "      <td>0.120</td>\n",
       "      <td>0.108</td>\n",
       "      <td>0.096</td>\n",
       "      <td>0.820</td>\n",
       "      <td>0.880</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G10-WW00-FSF10-DQLNet-MSELoss</th>\n",
       "      <td>0.552</td>\n",
       "      <td>0.436</td>\n",
       "      <td>0.832</td>\n",
       "      <td>0.136</td>\n",
       "      <td>0.196</td>\n",
       "      <td>0.148</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.892</td>\n",
       "      <td>0.076</td>\n",
       "      <td>0.824</td>\n",
       "      <td>0.068</td>\n",
       "      <td>0.796</td>\n",
       "      <td>0.712</td>\n",
       "      <td>0.160</td>\n",
       "      <td>0.760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW10-FSF00-DQLNet-MSELoss</th>\n",
       "      <td>0.660</td>\n",
       "      <td>0.492</td>\n",
       "      <td>0.784</td>\n",
       "      <td>0.792</td>\n",
       "      <td>0.848</td>\n",
       "      <td>0.196</td>\n",
       "      <td>0.800</td>\n",
       "      <td>0.104</td>\n",
       "      <td>0.144</td>\n",
       "      <td>0.808</td>\n",
       "      <td>0.752</td>\n",
       "      <td>0.120</td>\n",
       "      <td>0.088</td>\n",
       "      <td>0.784</td>\n",
       "      <td>0.752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW03-FSF03-DQLNet-MSELoss</th>\n",
       "      <td>0.560</td>\n",
       "      <td>0.496</td>\n",
       "      <td>0.728</td>\n",
       "      <td>0.044</td>\n",
       "      <td>0.800</td>\n",
       "      <td>0.128</td>\n",
       "      <td>0.088</td>\n",
       "      <td>0.812</td>\n",
       "      <td>0.940</td>\n",
       "      <td>0.168</td>\n",
       "      <td>0.884</td>\n",
       "      <td>0.132</td>\n",
       "      <td>0.156</td>\n",
       "      <td>0.876</td>\n",
       "      <td>0.044</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW03-FSF07-DQLNet-MSELoss</th>\n",
       "      <td>0.504</td>\n",
       "      <td>0.488</td>\n",
       "      <td>0.108</td>\n",
       "      <td>0.100</td>\n",
       "      <td>0.096</td>\n",
       "      <td>0.888</td>\n",
       "      <td>0.084</td>\n",
       "      <td>0.108</td>\n",
       "      <td>0.144</td>\n",
       "      <td>0.868</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.172</td>\n",
       "      <td>0.080</td>\n",
       "      <td>0.136</td>\n",
       "      <td>0.112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW02-FSF07-DQLNet-MSELoss</th>\n",
       "      <td>0.608</td>\n",
       "      <td>0.512</td>\n",
       "      <td>0.836</td>\n",
       "      <td>0.884</td>\n",
       "      <td>0.140</td>\n",
       "      <td>0.860</td>\n",
       "      <td>0.128</td>\n",
       "      <td>0.844</td>\n",
       "      <td>0.092</td>\n",
       "      <td>0.900</td>\n",
       "      <td>0.892</td>\n",
       "      <td>0.108</td>\n",
       "      <td>0.788</td>\n",
       "      <td>0.296</td>\n",
       "      <td>0.884</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "White policy                           random  greedy_policy  \\\n",
       "Black policy                                                   \n",
       "random                                  0.432          0.376   \n",
       "greedy_policy                           0.592          0.488   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss   0.592          0.456   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss   0.616          0.384   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss   0.580          0.532   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss   0.520          0.340   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss   0.628          0.616   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss   0.576          0.528   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss   0.524          0.404   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss      0.492          0.452   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss      0.552          0.436   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss      0.660          0.492   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss      0.560          0.496   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss      0.504          0.488   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss      0.608          0.512   \n",
       "\n",
       "White policy                           QL-M-G09-WW00-FSF10-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                 0.340   \n",
       "greedy_policy                                                          0.596   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                  0.856   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                  0.852   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                  0.796   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  0.820   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  0.756   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                  0.144   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                  0.876   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                     0.112   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                     0.832   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                     0.784   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                     0.728   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                     0.108   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     0.836   \n",
       "\n",
       "White policy                           QL-M-G08-WW00-FSF10-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                 0.312   \n",
       "greedy_policy                                                          0.392   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                  0.172   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                  0.792   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                  0.772   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  0.868   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  0.124   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                  0.800   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                  0.704   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                     0.128   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                     0.136   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                     0.792   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                     0.044   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                     0.100   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     0.884   \n",
       "\n",
       "White policy                           QL-M-G10-WW00-FSF10-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                 0.368   \n",
       "greedy_policy                                                          0.452   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                  0.748   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                  0.832   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                  0.804   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  0.080   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  0.204   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                  0.184   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                  0.088   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                     0.072   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                     0.196   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                     0.848   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                     0.800   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                     0.096   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     0.140   \n",
       "\n",
       "White policy                           QL-M-G09-WW10-FSF00-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                 0.344   \n",
       "greedy_policy                                                          0.544   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                  0.076   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                  0.160   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                  0.140   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  0.180   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  0.220   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                  0.108   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                  0.080   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                     0.844   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                     0.148   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                     0.196   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                     0.128   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                     0.888   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     0.860   \n",
       "\n",
       "White policy                           QL-M-G09-WW03-FSF03-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                 0.244   \n",
       "greedy_policy                                                          0.384   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                  0.136   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                  0.812   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                  0.172   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  0.140   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  0.204   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                  0.104   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                  0.064   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                     0.052   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                     0.060   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                     0.800   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                     0.088   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                     0.084   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     0.128   \n",
       "\n",
       "White policy                           QL-M-G09-WW03-FSF07-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                 0.332   \n",
       "greedy_policy                                                          0.436   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                  0.760   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                  0.112   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                  0.108   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  0.204   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  0.224   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                  0.256   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                  0.112   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                     0.088   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                     0.892   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                     0.104   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                     0.812   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                     0.108   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     0.844   \n",
       "\n",
       "White policy                           QL-M-G09-WW02-FSF07-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                 0.312   \n",
       "greedy_policy                                                          0.484   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                  0.176   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                  0.088   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                  0.888   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  0.780   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  0.872   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                  0.240   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                  0.128   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                     0.948   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                     0.076   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                     0.144   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                     0.940   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                     0.144   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     0.092   \n",
       "\n",
       "White policy                           QL-M-G08-WW00-FSF10-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                              0.452   \n",
       "greedy_policy                                                       0.536   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                               0.136   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                               0.224   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                               0.840   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                               0.716   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                               0.932   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                               0.164   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                               0.916   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                  0.196   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  0.824   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                  0.808   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                  0.168   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                  0.868   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                  0.900   \n",
       "\n",
       "White policy                           QL-M-G10-WW00-FSF10-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                              0.372   \n",
       "greedy_policy                                                       0.472   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                               0.104   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                               0.744   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                               0.700   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                               0.876   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                               0.824   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                               0.140   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                               0.192   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                  0.120   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  0.068   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                  0.752   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                  0.884   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                  0.060   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                  0.892   \n",
       "\n",
       "White policy                           QL-M-G09-WW10-FSF00-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                              0.272   \n",
       "greedy_policy                                                       0.432   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                               0.096   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                               0.076   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                               0.104   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                               0.100   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                               0.136   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                               0.108   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                               0.776   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                  0.108   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  0.796   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                  0.120   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                  0.132   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                  0.172   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                  0.108   \n",
       "\n",
       "White policy                           QL-M-G09-WW03-FSF03-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                              0.352   \n",
       "greedy_policy                                                       0.580   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                               0.804   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                               0.828   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                               0.720   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                               0.088   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                               0.080   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                               0.096   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                               0.220   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                  0.096   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  0.712   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                  0.088   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                  0.156   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                  0.080   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                  0.788   \n",
       "\n",
       "White policy                           QL-M-G09-WW03-FSF07-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                              0.388   \n",
       "greedy_policy                                                       0.508   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                               0.908   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                               0.756   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                               0.188   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                               0.800   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                               0.236   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                               0.180   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                               0.720   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                  0.820   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  0.160   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                  0.784   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                  0.876   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                  0.136   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                  0.296   \n",
       "\n",
       "White policy                           QL-M-G09-WW02-FSF07-DQLNet-MSELoss  \n",
       "Black policy                                                               \n",
       "random                                                              0.356  \n",
       "greedy_policy                                                       0.444  \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                               0.720  \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                               0.108  \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                               0.744  \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                               0.176  \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                               0.208  \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                               0.756  \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                               0.908  \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                  0.880  \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  0.760  \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                  0.752  \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                  0.044  \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                  0.112  \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                  0.884  "
      ]
     },
     "execution_count": 207,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = result_df.applymap(lambda x: x[\"win\"])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To mitigate the bias in the matrix, we calculate a perspective-correct mean by subtracting its transposed version and dividing by two. This compensates for the black policy's relative advantage. However, the resulting values may not all be positive, as some policies may now have a negative average win signum. Specifically, a policy with a win signum of 0.1 will win 60% of the games.\n",
    "\n",
    "Despite this, we observe that learning did occur, as most games played against the random and greedy policies resulted in a positive score. However, the degree of learning fell far short of our expectations. A simple policy that weights each stone by its position may be much more effective, particularly if the value trends towards 1 towards the end of the game. This aligns with our earlier observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>White policy</th>\n",
       "      <th>random</th>\n",
       "      <th>greedy_policy</th>\n",
       "      <th>QL-M-G09-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G08-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G10-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G09-WW10-FSF00-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G09-WW03-FSF03-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G09-WW03-FSF07-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G09-WW02-FSF07-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G08-WW00-FSF10-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G10-WW00-FSF10-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G09-WW10-FSF00-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G09-WW03-FSF03-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G09-WW03-FSF07-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G09-WW02-FSF07-DQLNet-MSELoss</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Black policy</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>random</th>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.108</td>\n",
       "      <td>-0.126</td>\n",
       "      <td>-0.152</td>\n",
       "      <td>-0.106</td>\n",
       "      <td>-0.088</td>\n",
       "      <td>-0.192</td>\n",
       "      <td>-0.122</td>\n",
       "      <td>-0.106</td>\n",
       "      <td>-0.020</td>\n",
       "      <td>-0.090</td>\n",
       "      <td>-0.194</td>\n",
       "      <td>-0.104</td>\n",
       "      <td>-0.058</td>\n",
       "      <td>-0.126</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>greedy_policy</th>\n",
       "      <td>0.108</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.070</td>\n",
       "      <td>0.004</td>\n",
       "      <td>-0.040</td>\n",
       "      <td>0.102</td>\n",
       "      <td>-0.116</td>\n",
       "      <td>-0.046</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.042</td>\n",
       "      <td>0.018</td>\n",
       "      <td>-0.030</td>\n",
       "      <td>0.042</td>\n",
       "      <td>0.010</td>\n",
       "      <td>-0.034</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <td>0.126</td>\n",
       "      <td>-0.070</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.340</td>\n",
       "      <td>-0.024</td>\n",
       "      <td>-0.372</td>\n",
       "      <td>-0.310</td>\n",
       "      <td>0.308</td>\n",
       "      <td>-0.350</td>\n",
       "      <td>0.012</td>\n",
       "      <td>-0.364</td>\n",
       "      <td>-0.344</td>\n",
       "      <td>0.038</td>\n",
       "      <td>0.400</td>\n",
       "      <td>-0.058</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G08-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <td>0.152</td>\n",
       "      <td>-0.004</td>\n",
       "      <td>0.340</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.030</td>\n",
       "      <td>-0.354</td>\n",
       "      <td>0.344</td>\n",
       "      <td>-0.344</td>\n",
       "      <td>-0.308</td>\n",
       "      <td>0.048</td>\n",
       "      <td>0.304</td>\n",
       "      <td>-0.358</td>\n",
       "      <td>0.392</td>\n",
       "      <td>0.328</td>\n",
       "      <td>-0.388</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G10-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <td>0.106</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.024</td>\n",
       "      <td>-0.030</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.030</td>\n",
       "      <td>-0.016</td>\n",
       "      <td>-0.038</td>\n",
       "      <td>0.400</td>\n",
       "      <td>0.384</td>\n",
       "      <td>0.252</td>\n",
       "      <td>-0.372</td>\n",
       "      <td>-0.040</td>\n",
       "      <td>0.046</td>\n",
       "      <td>0.302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW10-FSF00-DQLSimple-MSELoss</th>\n",
       "      <td>0.088</td>\n",
       "      <td>-0.102</td>\n",
       "      <td>0.372</td>\n",
       "      <td>0.354</td>\n",
       "      <td>-0.030</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.040</td>\n",
       "      <td>0.048</td>\n",
       "      <td>0.350</td>\n",
       "      <td>-0.064</td>\n",
       "      <td>0.364</td>\n",
       "      <td>-0.048</td>\n",
       "      <td>-0.020</td>\n",
       "      <td>-0.044</td>\n",
       "      <td>-0.342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW03-FSF03-DQLSimple-MSELoss</th>\n",
       "      <td>0.192</td>\n",
       "      <td>0.116</td>\n",
       "      <td>0.310</td>\n",
       "      <td>-0.344</td>\n",
       "      <td>0.016</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.060</td>\n",
       "      <td>0.404</td>\n",
       "      <td>0.440</td>\n",
       "      <td>0.382</td>\n",
       "      <td>-0.332</td>\n",
       "      <td>-0.004</td>\n",
       "      <td>0.076</td>\n",
       "      <td>0.040</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW03-FSF07-DQLSimple-MSELoss</th>\n",
       "      <td>0.122</td>\n",
       "      <td>0.046</td>\n",
       "      <td>-0.308</td>\n",
       "      <td>0.344</td>\n",
       "      <td>0.038</td>\n",
       "      <td>-0.048</td>\n",
       "      <td>-0.060</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.064</td>\n",
       "      <td>0.038</td>\n",
       "      <td>-0.376</td>\n",
       "      <td>0.002</td>\n",
       "      <td>-0.358</td>\n",
       "      <td>0.036</td>\n",
       "      <td>-0.044</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW02-FSF07-DQLSimple-MSELoss</th>\n",
       "      <td>0.106</td>\n",
       "      <td>-0.040</td>\n",
       "      <td>0.350</td>\n",
       "      <td>0.308</td>\n",
       "      <td>-0.400</td>\n",
       "      <td>-0.350</td>\n",
       "      <td>-0.404</td>\n",
       "      <td>-0.064</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.016</td>\n",
       "      <td>0.058</td>\n",
       "      <td>0.316</td>\n",
       "      <td>-0.360</td>\n",
       "      <td>0.288</td>\n",
       "      <td>0.408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G08-WW00-FSF10-DQLNet-MSELoss</th>\n",
       "      <td>0.020</td>\n",
       "      <td>-0.042</td>\n",
       "      <td>-0.012</td>\n",
       "      <td>-0.048</td>\n",
       "      <td>-0.384</td>\n",
       "      <td>0.064</td>\n",
       "      <td>-0.440</td>\n",
       "      <td>-0.038</td>\n",
       "      <td>0.016</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.352</td>\n",
       "      <td>-0.350</td>\n",
       "      <td>-0.036</td>\n",
       "      <td>-0.024</td>\n",
       "      <td>-0.010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G10-WW00-FSF10-DQLNet-MSELoss</th>\n",
       "      <td>0.090</td>\n",
       "      <td>-0.018</td>\n",
       "      <td>0.364</td>\n",
       "      <td>-0.304</td>\n",
       "      <td>-0.252</td>\n",
       "      <td>-0.364</td>\n",
       "      <td>-0.382</td>\n",
       "      <td>0.376</td>\n",
       "      <td>-0.058</td>\n",
       "      <td>0.352</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.022</td>\n",
       "      <td>-0.086</td>\n",
       "      <td>0.050</td>\n",
       "      <td>-0.066</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW10-FSF00-DQLNet-MSELoss</th>\n",
       "      <td>0.194</td>\n",
       "      <td>0.030</td>\n",
       "      <td>0.344</td>\n",
       "      <td>0.358</td>\n",
       "      <td>0.372</td>\n",
       "      <td>0.048</td>\n",
       "      <td>0.332</td>\n",
       "      <td>-0.002</td>\n",
       "      <td>-0.316</td>\n",
       "      <td>0.350</td>\n",
       "      <td>-0.022</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.022</td>\n",
       "      <td>0.306</td>\n",
       "      <td>0.322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW03-FSF03-DQLNet-MSELoss</th>\n",
       "      <td>0.104</td>\n",
       "      <td>-0.042</td>\n",
       "      <td>-0.038</td>\n",
       "      <td>-0.392</td>\n",
       "      <td>0.040</td>\n",
       "      <td>0.020</td>\n",
       "      <td>0.004</td>\n",
       "      <td>0.358</td>\n",
       "      <td>0.360</td>\n",
       "      <td>0.036</td>\n",
       "      <td>0.086</td>\n",
       "      <td>0.022</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.398</td>\n",
       "      <td>-0.372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW03-FSF07-DQLNet-MSELoss</th>\n",
       "      <td>0.058</td>\n",
       "      <td>-0.010</td>\n",
       "      <td>-0.400</td>\n",
       "      <td>-0.328</td>\n",
       "      <td>-0.046</td>\n",
       "      <td>0.044</td>\n",
       "      <td>-0.076</td>\n",
       "      <td>-0.036</td>\n",
       "      <td>-0.288</td>\n",
       "      <td>0.024</td>\n",
       "      <td>-0.050</td>\n",
       "      <td>-0.306</td>\n",
       "      <td>-0.398</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-0.092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW02-FSF07-DQLNet-MSELoss</th>\n",
       "      <td>0.126</td>\n",
       "      <td>0.034</td>\n",
       "      <td>0.058</td>\n",
       "      <td>0.388</td>\n",
       "      <td>-0.302</td>\n",
       "      <td>0.342</td>\n",
       "      <td>-0.040</td>\n",
       "      <td>0.044</td>\n",
       "      <td>-0.408</td>\n",
       "      <td>0.010</td>\n",
       "      <td>0.066</td>\n",
       "      <td>-0.322</td>\n",
       "      <td>0.372</td>\n",
       "      <td>0.092</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "White policy                           random  greedy_policy  \\\n",
       "Black policy                                                   \n",
       "random                                  0.000         -0.108   \n",
       "greedy_policy                           0.108          0.000   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss   0.126         -0.070   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss   0.152         -0.004   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss   0.106          0.040   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss   0.088         -0.102   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss   0.192          0.116   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss   0.122          0.046   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss   0.106         -0.040   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss      0.020         -0.042   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss      0.090         -0.018   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss      0.194          0.030   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss      0.104         -0.042   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss      0.058         -0.010   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss      0.126          0.034   \n",
       "\n",
       "White policy                           QL-M-G09-WW00-FSF10-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                -0.126   \n",
       "greedy_policy                                                          0.070   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                  0.000   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                  0.340   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                  0.024   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  0.372   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  0.310   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                 -0.308   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                  0.350   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                    -0.012   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                     0.364   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                     0.344   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                    -0.038   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                    -0.400   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     0.058   \n",
       "\n",
       "White policy                           QL-M-G08-WW00-FSF10-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                -0.152   \n",
       "greedy_policy                                                          0.004   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                 -0.340   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                  0.000   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                 -0.030   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  0.354   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                 -0.344   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                  0.344   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                  0.308   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                    -0.048   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                    -0.304   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                     0.358   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                    -0.392   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                    -0.328   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     0.388   \n",
       "\n",
       "White policy                           QL-M-G10-WW00-FSF10-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                -0.106   \n",
       "greedy_policy                                                         -0.040   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                 -0.024   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                  0.030   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                  0.000   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                 -0.030   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  0.016   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                  0.038   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                 -0.400   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                    -0.384   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                    -0.252   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                     0.372   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                     0.040   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                    -0.046   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                    -0.302   \n",
       "\n",
       "White policy                           QL-M-G09-WW10-FSF00-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                -0.088   \n",
       "greedy_policy                                                          0.102   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                 -0.372   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                 -0.354   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                  0.030   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  0.000   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  0.040   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                 -0.048   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                 -0.350   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                     0.064   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                    -0.364   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                     0.048   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                     0.020   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                     0.044   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     0.342   \n",
       "\n",
       "White policy                           QL-M-G09-WW03-FSF03-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                -0.192   \n",
       "greedy_policy                                                         -0.116   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                 -0.310   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                  0.344   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                 -0.016   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                 -0.040   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  0.000   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                 -0.060   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                 -0.404   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                    -0.440   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                    -0.382   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                     0.332   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                     0.004   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                    -0.076   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                    -0.040   \n",
       "\n",
       "White policy                           QL-M-G09-WW03-FSF07-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                -0.122   \n",
       "greedy_policy                                                         -0.046   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                  0.308   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                 -0.344   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                 -0.038   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  0.048   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  0.060   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                  0.000   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                 -0.064   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                    -0.038   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                     0.376   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                    -0.002   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                     0.358   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                    -0.036   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     0.044   \n",
       "\n",
       "White policy                           QL-M-G09-WW02-FSF07-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                -0.106   \n",
       "greedy_policy                                                          0.040   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                 -0.350   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                 -0.308   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                  0.400   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  0.350   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  0.404   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                  0.064   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                  0.000   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                     0.016   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                    -0.058   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                    -0.316   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                     0.360   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                    -0.288   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                    -0.408   \n",
       "\n",
       "White policy                           QL-M-G08-WW00-FSF10-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                             -0.020   \n",
       "greedy_policy                                                       0.042   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                               0.012   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                               0.048   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                               0.384   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                              -0.064   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                               0.440   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                               0.038   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                              -0.016   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                  0.000   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  0.352   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                  0.350   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                  0.036   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                  0.024   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                  0.010   \n",
       "\n",
       "White policy                           QL-M-G10-WW00-FSF10-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                             -0.090   \n",
       "greedy_policy                                                       0.018   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                              -0.364   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                               0.304   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                               0.252   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                               0.364   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                               0.382   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                              -0.376   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                               0.058   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                 -0.352   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  0.000   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                 -0.022   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                  0.086   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                 -0.050   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                  0.066   \n",
       "\n",
       "White policy                           QL-M-G09-WW10-FSF00-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                             -0.194   \n",
       "greedy_policy                                                      -0.030   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                              -0.344   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                              -0.358   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                              -0.372   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                              -0.048   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                              -0.332   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                               0.002   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                               0.316   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                 -0.350   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  0.022   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                  0.000   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                  0.022   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                 -0.306   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                 -0.322   \n",
       "\n",
       "White policy                           QL-M-G09-WW03-FSF03-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                             -0.104   \n",
       "greedy_policy                                                       0.042   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                               0.038   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                               0.392   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                              -0.040   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                              -0.020   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                              -0.004   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                              -0.358   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                              -0.360   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                 -0.036   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                 -0.086   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                 -0.022   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                  0.000   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                 -0.398   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                  0.372   \n",
       "\n",
       "White policy                           QL-M-G09-WW03-FSF07-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                             -0.058   \n",
       "greedy_policy                                                       0.010   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                               0.400   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                               0.328   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                               0.046   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                              -0.044   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                               0.076   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                               0.036   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                               0.288   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                 -0.024   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  0.050   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                  0.306   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                  0.398   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                  0.000   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                  0.092   \n",
       "\n",
       "White policy                           QL-M-G09-WW02-FSF07-DQLNet-MSELoss  \n",
       "Black policy                                                               \n",
       "random                                                             -0.126  \n",
       "greedy_policy                                                      -0.034  \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                              -0.058  \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                              -0.388  \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                               0.302  \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                              -0.342  \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                               0.040  \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                              -0.044  \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                               0.408  \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                 -0.010  \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                 -0.066  \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                  0.322  \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                 -0.372  \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                 -0.092  \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                  0.000  "
      ]
     },
     "execution_count": 208,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = result_df.applymap(lambda x: x[\"win\"])\n",
    "(df - df.T) / 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sadly there is no clear best policy to identify."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysing the polices by the final score\n",
    "When analysing the final score it is also not easy to see any QL-policy clearly outpacing the other QL-policies.\n",
    "Score color bias compensation is done the same as above. The multiplication by 64 compensates for the score beeing normed to [1, -1]. The score now calculates the advantage in points a player has."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 327,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>White policy</th>\n",
       "      <th>random</th>\n",
       "      <th>greedy_policy</th>\n",
       "      <th>QL-M-G09-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G08-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G10-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G09-WW10-FSF00-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G09-WW03-FSF03-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G09-WW03-FSF07-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G09-WW02-FSF07-DQLSimple-MSELoss</th>\n",
       "      <th>QL-M-G08-WW00-FSF10-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G10-WW00-FSF10-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G09-WW10-FSF00-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G09-WW03-FSF03-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G09-WW03-FSF07-DQLNet-MSELoss</th>\n",
       "      <th>QL-M-G09-WW02-FSF07-DQLNet-MSELoss</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Black policy</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>random</th>\n",
       "      <td>0.000</td>\n",
       "      <td>-7.828</td>\n",
       "      <td>-6.020</td>\n",
       "      <td>-6.652</td>\n",
       "      <td>-3.832</td>\n",
       "      <td>-3.400</td>\n",
       "      <td>-8.016</td>\n",
       "      <td>-6.296</td>\n",
       "      <td>-5.636</td>\n",
       "      <td>-0.800</td>\n",
       "      <td>-4.208</td>\n",
       "      <td>-8.928</td>\n",
       "      <td>-5.372</td>\n",
       "      <td>-2.976</td>\n",
       "      <td>-6.084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>greedy_policy</th>\n",
       "      <td>7.828</td>\n",
       "      <td>0.000</td>\n",
       "      <td>4.028</td>\n",
       "      <td>3.084</td>\n",
       "      <td>0.472</td>\n",
       "      <td>7.428</td>\n",
       "      <td>-2.200</td>\n",
       "      <td>-0.392</td>\n",
       "      <td>2.752</td>\n",
       "      <td>4.560</td>\n",
       "      <td>2.292</td>\n",
       "      <td>0.216</td>\n",
       "      <td>4.748</td>\n",
       "      <td>1.404</td>\n",
       "      <td>0.240</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <td>6.020</td>\n",
       "      <td>-4.028</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-14.176</td>\n",
       "      <td>-2.912</td>\n",
       "      <td>-11.464</td>\n",
       "      <td>-5.176</td>\n",
       "      <td>9.160</td>\n",
       "      <td>-12.136</td>\n",
       "      <td>0.924</td>\n",
       "      <td>-14.460</td>\n",
       "      <td>-13.608</td>\n",
       "      <td>1.912</td>\n",
       "      <td>13.204</td>\n",
       "      <td>-3.764</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G08-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <td>6.652</td>\n",
       "      <td>-3.084</td>\n",
       "      <td>14.176</td>\n",
       "      <td>0.000</td>\n",
       "      <td>3.460</td>\n",
       "      <td>-7.600</td>\n",
       "      <td>10.820</td>\n",
       "      <td>-20.352</td>\n",
       "      <td>-9.228</td>\n",
       "      <td>12.092</td>\n",
       "      <td>12.940</td>\n",
       "      <td>-13.204</td>\n",
       "      <td>12.212</td>\n",
       "      <td>12.672</td>\n",
       "      <td>-16.348</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G10-WW00-FSF10-DQLSimple-MSELoss</th>\n",
       "      <td>3.832</td>\n",
       "      <td>-0.472</td>\n",
       "      <td>2.912</td>\n",
       "      <td>-3.460</td>\n",
       "      <td>0.000</td>\n",
       "      <td>9.660</td>\n",
       "      <td>-4.916</td>\n",
       "      <td>-7.840</td>\n",
       "      <td>17.628</td>\n",
       "      <td>11.796</td>\n",
       "      <td>-1.080</td>\n",
       "      <td>-17.580</td>\n",
       "      <td>-15.264</td>\n",
       "      <td>6.168</td>\n",
       "      <td>9.144</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW10-FSF00-DQLSimple-MSELoss</th>\n",
       "      <td>3.400</td>\n",
       "      <td>-7.428</td>\n",
       "      <td>11.464</td>\n",
       "      <td>7.600</td>\n",
       "      <td>-9.660</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-1.240</td>\n",
       "      <td>1.832</td>\n",
       "      <td>8.036</td>\n",
       "      <td>1.028</td>\n",
       "      <td>14.328</td>\n",
       "      <td>-8.756</td>\n",
       "      <td>1.544</td>\n",
       "      <td>-1.692</td>\n",
       "      <td>-9.496</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW03-FSF03-DQLSimple-MSELoss</th>\n",
       "      <td>8.016</td>\n",
       "      <td>2.200</td>\n",
       "      <td>5.176</td>\n",
       "      <td>-10.820</td>\n",
       "      <td>4.916</td>\n",
       "      <td>1.240</td>\n",
       "      <td>0.000</td>\n",
       "      <td>2.340</td>\n",
       "      <td>11.268</td>\n",
       "      <td>13.256</td>\n",
       "      <td>12.028</td>\n",
       "      <td>-7.292</td>\n",
       "      <td>-8.536</td>\n",
       "      <td>11.360</td>\n",
       "      <td>-7.336</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW03-FSF07-DQLSimple-MSELoss</th>\n",
       "      <td>6.296</td>\n",
       "      <td>0.392</td>\n",
       "      <td>-9.160</td>\n",
       "      <td>20.352</td>\n",
       "      <td>7.840</td>\n",
       "      <td>-1.832</td>\n",
       "      <td>-2.340</td>\n",
       "      <td>0.000</td>\n",
       "      <td>5.020</td>\n",
       "      <td>7.676</td>\n",
       "      <td>-7.772</td>\n",
       "      <td>4.564</td>\n",
       "      <td>-9.492</td>\n",
       "      <td>0.448</td>\n",
       "      <td>-2.552</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW02-FSF07-DQLSimple-MSELoss</th>\n",
       "      <td>5.636</td>\n",
       "      <td>-2.752</td>\n",
       "      <td>12.136</td>\n",
       "      <td>9.228</td>\n",
       "      <td>-17.628</td>\n",
       "      <td>-8.036</td>\n",
       "      <td>-11.268</td>\n",
       "      <td>-5.020</td>\n",
       "      <td>0.000</td>\n",
       "      <td>7.720</td>\n",
       "      <td>9.916</td>\n",
       "      <td>12.032</td>\n",
       "      <td>-24.100</td>\n",
       "      <td>9.612</td>\n",
       "      <td>18.056</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G08-WW00-FSF10-DQLNet-MSELoss</th>\n",
       "      <td>0.800</td>\n",
       "      <td>-4.560</td>\n",
       "      <td>-0.924</td>\n",
       "      <td>-12.092</td>\n",
       "      <td>-11.796</td>\n",
       "      <td>-1.028</td>\n",
       "      <td>-13.256</td>\n",
       "      <td>-7.676</td>\n",
       "      <td>-7.720</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-8.476</td>\n",
       "      <td>-8.544</td>\n",
       "      <td>-6.116</td>\n",
       "      <td>-1.380</td>\n",
       "      <td>14.544</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G10-WW00-FSF10-DQLNet-MSELoss</th>\n",
       "      <td>4.208</td>\n",
       "      <td>-2.292</td>\n",
       "      <td>14.460</td>\n",
       "      <td>-12.940</td>\n",
       "      <td>1.080</td>\n",
       "      <td>-14.328</td>\n",
       "      <td>-12.028</td>\n",
       "      <td>7.772</td>\n",
       "      <td>-9.916</td>\n",
       "      <td>8.476</td>\n",
       "      <td>0.000</td>\n",
       "      <td>1.476</td>\n",
       "      <td>-1.876</td>\n",
       "      <td>5.328</td>\n",
       "      <td>2.228</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW10-FSF00-DQLNet-MSELoss</th>\n",
       "      <td>8.928</td>\n",
       "      <td>-0.216</td>\n",
       "      <td>13.608</td>\n",
       "      <td>13.204</td>\n",
       "      <td>17.580</td>\n",
       "      <td>8.756</td>\n",
       "      <td>7.292</td>\n",
       "      <td>-4.564</td>\n",
       "      <td>-12.032</td>\n",
       "      <td>8.544</td>\n",
       "      <td>-1.476</td>\n",
       "      <td>0.000</td>\n",
       "      <td>5.768</td>\n",
       "      <td>15.268</td>\n",
       "      <td>18.408</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW03-FSF03-DQLNet-MSELoss</th>\n",
       "      <td>5.372</td>\n",
       "      <td>-4.748</td>\n",
       "      <td>-1.912</td>\n",
       "      <td>-12.212</td>\n",
       "      <td>15.264</td>\n",
       "      <td>-1.544</td>\n",
       "      <td>8.536</td>\n",
       "      <td>9.492</td>\n",
       "      <td>24.100</td>\n",
       "      <td>6.116</td>\n",
       "      <td>1.876</td>\n",
       "      <td>-5.768</td>\n",
       "      <td>0.000</td>\n",
       "      <td>27.580</td>\n",
       "      <td>-14.512</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW03-FSF07-DQLNet-MSELoss</th>\n",
       "      <td>2.976</td>\n",
       "      <td>-1.404</td>\n",
       "      <td>-13.204</td>\n",
       "      <td>-12.672</td>\n",
       "      <td>-6.168</td>\n",
       "      <td>1.692</td>\n",
       "      <td>-11.360</td>\n",
       "      <td>-0.448</td>\n",
       "      <td>-9.612</td>\n",
       "      <td>1.380</td>\n",
       "      <td>-5.328</td>\n",
       "      <td>-15.268</td>\n",
       "      <td>-27.580</td>\n",
       "      <td>0.000</td>\n",
       "      <td>-1.792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>QL-M-G09-WW02-FSF07-DQLNet-MSELoss</th>\n",
       "      <td>6.084</td>\n",
       "      <td>-0.240</td>\n",
       "      <td>3.764</td>\n",
       "      <td>16.348</td>\n",
       "      <td>-9.144</td>\n",
       "      <td>9.496</td>\n",
       "      <td>7.336</td>\n",
       "      <td>2.552</td>\n",
       "      <td>-18.056</td>\n",
       "      <td>-14.544</td>\n",
       "      <td>-2.228</td>\n",
       "      <td>-18.408</td>\n",
       "      <td>14.512</td>\n",
       "      <td>1.792</td>\n",
       "      <td>0.000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "White policy                           random  greedy_policy  \\\n",
       "Black policy                                                   \n",
       "random                                  0.000         -7.828   \n",
       "greedy_policy                           7.828          0.000   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss   6.020         -4.028   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss   6.652         -3.084   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss   3.832         -0.472   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss   3.400         -7.428   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss   8.016          2.200   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss   6.296          0.392   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss   5.636         -2.752   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss      0.800         -4.560   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss      4.208         -2.292   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss      8.928         -0.216   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss      5.372         -4.748   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss      2.976         -1.404   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss      6.084         -0.240   \n",
       "\n",
       "White policy                           QL-M-G09-WW00-FSF10-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                -6.020   \n",
       "greedy_policy                                                          4.028   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                  0.000   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                 14.176   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                  2.912   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                 11.464   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  5.176   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                 -9.160   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                 12.136   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                    -0.924   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                    14.460   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                    13.608   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                    -1.912   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                   -13.204   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     3.764   \n",
       "\n",
       "White policy                           QL-M-G08-WW00-FSF10-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                -6.652   \n",
       "greedy_policy                                                          3.084   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                -14.176   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                  0.000   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                 -3.460   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  7.600   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                -10.820   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                 20.352   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                  9.228   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                   -12.092   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                   -12.940   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                    13.204   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                   -12.212   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                   -12.672   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                    16.348   \n",
       "\n",
       "White policy                           QL-M-G10-WW00-FSF10-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                -3.832   \n",
       "greedy_policy                                                          0.472   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                 -2.912   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                  3.460   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                  0.000   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                 -9.660   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  4.916   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                  7.840   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                -17.628   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                   -11.796   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                     1.080   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                    17.580   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                    15.264   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                    -6.168   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                    -9.144   \n",
       "\n",
       "White policy                           QL-M-G09-WW10-FSF00-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                -3.400   \n",
       "greedy_policy                                                          7.428   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                -11.464   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                 -7.600   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                  9.660   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  0.000   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  1.240   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                 -1.832   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                 -8.036   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                    -1.028   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                   -14.328   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                     8.756   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                    -1.544   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                     1.692   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     9.496   \n",
       "\n",
       "White policy                           QL-M-G09-WW03-FSF03-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                -8.016   \n",
       "greedy_policy                                                         -2.200   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                 -5.176   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                 10.820   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                 -4.916   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                 -1.240   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  0.000   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                 -2.340   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                -11.268   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                   -13.256   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                   -12.028   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                     7.292   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                     8.536   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                   -11.360   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     7.336   \n",
       "\n",
       "White policy                           QL-M-G09-WW03-FSF07-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                -6.296   \n",
       "greedy_policy                                                         -0.392   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                  9.160   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                -20.352   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                 -7.840   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  1.832   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                  2.340   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                  0.000   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                 -5.020   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                    -7.676   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                     7.772   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                    -4.564   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                     9.492   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                    -0.448   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                     2.552   \n",
       "\n",
       "White policy                           QL-M-G09-WW02-FSF07-DQLSimple-MSELoss  \\\n",
       "Black policy                                                                   \n",
       "random                                                                -5.636   \n",
       "greedy_policy                                                          2.752   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                                -12.136   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                                 -9.228   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                                 17.628   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                                  8.036   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                                 11.268   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                                  5.020   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                                  0.000   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                    -7.720   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                    -9.916   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                   -12.032   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                    24.100   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                    -9.612   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                   -18.056   \n",
       "\n",
       "White policy                           QL-M-G08-WW00-FSF10-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                             -0.800   \n",
       "greedy_policy                                                       4.560   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                               0.924   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                              12.092   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                              11.796   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                               1.028   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                              13.256   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                               7.676   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                               7.720   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                  0.000   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  8.476   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                  8.544   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                  6.116   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                  1.380   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                -14.544   \n",
       "\n",
       "White policy                           QL-M-G10-WW00-FSF10-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                             -4.208   \n",
       "greedy_policy                                                       2.292   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                             -14.460   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                              12.940   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                              -1.080   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                              14.328   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                              12.028   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                              -7.772   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                               9.916   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                 -8.476   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  0.000   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                 -1.476   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                  1.876   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                 -5.328   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                 -2.228   \n",
       "\n",
       "White policy                           QL-M-G09-WW10-FSF00-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                             -8.928   \n",
       "greedy_policy                                                       0.216   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                             -13.608   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                             -13.204   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                             -17.580   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                              -8.756   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                              -7.292   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                               4.564   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                              12.032   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                 -8.544   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  1.476   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                  0.000   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                 -5.768   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                -15.268   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                -18.408   \n",
       "\n",
       "White policy                           QL-M-G09-WW03-FSF03-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                             -5.372   \n",
       "greedy_policy                                                       4.748   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                               1.912   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                              12.212   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                             -15.264   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                               1.544   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                              -8.536   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                              -9.492   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                             -24.100   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                 -6.116   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                 -1.876   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                  5.768   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                  0.000   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                -27.580   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                 14.512   \n",
       "\n",
       "White policy                           QL-M-G09-WW03-FSF07-DQLNet-MSELoss  \\\n",
       "Black policy                                                                \n",
       "random                                                             -2.976   \n",
       "greedy_policy                                                       1.404   \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                              13.204   \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                              12.672   \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                               6.168   \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                              -1.692   \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                              11.360   \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                               0.448   \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                               9.612   \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                 -1.380   \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  5.328   \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                 15.268   \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                 27.580   \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                  0.000   \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                  1.792   \n",
       "\n",
       "White policy                           QL-M-G09-WW02-FSF07-DQLNet-MSELoss  \n",
       "Black policy                                                               \n",
       "random                                                             -6.084  \n",
       "greedy_policy                                                       0.240  \n",
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss                              -3.764  \n",
       "QL-M-G08-WW00-FSF10-DQLSimple-MSELoss                             -16.348  \n",
       "QL-M-G10-WW00-FSF10-DQLSimple-MSELoss                               9.144  \n",
       "QL-M-G09-WW10-FSF00-DQLSimple-MSELoss                              -9.496  \n",
       "QL-M-G09-WW03-FSF03-DQLSimple-MSELoss                              -7.336  \n",
       "QL-M-G09-WW03-FSF07-DQLSimple-MSELoss                              -2.552  \n",
       "QL-M-G09-WW02-FSF07-DQLSimple-MSELoss                              18.056  \n",
       "QL-M-G08-WW00-FSF10-DQLNet-MSELoss                                 14.544  \n",
       "QL-M-G10-WW00-FSF10-DQLNet-MSELoss                                  2.228  \n",
       "QL-M-G09-WW10-FSF00-DQLNet-MSELoss                                 18.408  \n",
       "QL-M-G09-WW03-FSF03-DQLNet-MSELoss                                -14.512  \n",
       "QL-M-G09-WW03-FSF07-DQLNet-MSELoss                                 -1.792  \n",
       "QL-M-G09-WW02-FSF07-DQLNet-MSELoss                                  0.000  "
      ]
     },
     "execution_count": 327,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = result_df.applymap(lambda x: x[\"final_score\"])\n",
    "(df - df.T) / 2 * 64"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is no clear best policy to identify."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysing the corner behavior\n",
    "\n",
    "Given the importance of corner capture in Othello strategies, it is worth investigating if the Policies discover this on their own. To analyze this behavior, we selected a policy and measured the availability of corners, the probability of a corner stone being available, and the number of corners captured by the AI when playing against a random player."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QL-M-G09-WW00-FSF10-DQLSimple-MSELoss"
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_policy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "050e6c7cd5f64df181ea27364da2b52d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/70 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2f01a39831c641929b5a2d02559e38d6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/70 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "_l_board_history, _l_action_history = simulate_game(\n",
    "    1000, (example_policy, RandomPolicy(1)), True\n",
    ")\n",
    "_r_board_history, _r_action_history = simulate_game(\n",
    "    1000, (RandomPolicy(1), example_policy), True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((70, 8, 8), (70, 8, 8))"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_l_actions_possible = get_possible_turns(_l_board_history.reshape(-1, 8, 8)).reshape(\n",
    "    (70, -1, 8, 8)\n",
    ")\n",
    "_r_actions_possible = get_possible_turns(_r_board_history.reshape(-1, 8, 8)).reshape(\n",
    "    (70, -1, 8, 8)\n",
    ")\n",
    "\n",
    "\n",
    "_l_mean_actions_possible = np.mean(_l_actions_possible, axis=1)\n",
    "_r_mean_actions_possible = np.mean(_r_actions_possible, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Corner stone analysis via possible action space\n",
    "As noted in the analysis of the game and its behavior over time a heat map can be used to calculate how a game was won compared to a random strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "93d3ac66a66647969b281bef96207d38",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(IntSlider(value=34, description='turn', max=69), Output()), _dom_classes=('widget-intera…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "@interact(turn=(0, 69))\n",
    "def turn_distribution_heatmap(turn):\n",
    "    fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15, 4))\n",
    "    l_turn_possibility_on_field = _l_mean_actions_possible[turn]\n",
    "    r_turn_possibility_on_field = _r_mean_actions_possible[turn]\n",
    "    turn_possibility_on_field = mean_poss_turn[turn]\n",
    "\n",
    "    vmax = max(\n",
    "        np.max(l_turn_possibility_on_field),\n",
    "        np.max(r_turn_possibility_on_field),\n",
    "        np.max(turn_possibility_on_field),\n",
    "    )\n",
    "\n",
    "    sns.heatmap(\n",
    "        turn_possibility_on_field,\n",
    "        linewidth=0.5,\n",
    "        square=True,\n",
    "        annot=True,\n",
    "        xticklabels=\"ABCDEFGH\",\n",
    "        yticklabels=list(range(1, 9)),\n",
    "        ax=ax2,\n",
    "        vmax=vmax,\n",
    "    )\n",
    "    ax2.set_title(\"For a random player\")\n",
    "\n",
    "    sns.heatmap(\n",
    "        l_turn_possibility_on_field,\n",
    "        linewidth=0.5,\n",
    "        square=True,\n",
    "        annot=True,\n",
    "        xticklabels=\"ABCDEFGH\",\n",
    "        yticklabels=list(range(1, 9)),\n",
    "        ax=ax1,\n",
    "        vmax=vmax,\n",
    "    )\n",
    "    ax1.set_title(\"The trained policy for the player (black)\")\n",
    "\n",
    "    sns.heatmap(\n",
    "        r_turn_possibility_on_field,\n",
    "        linewidth=0.5,\n",
    "        square=True,\n",
    "        annot=True,\n",
    "        xticklabels=\"ABCDEFGH\",\n",
    "        yticklabels=list(range(1, 9)),\n",
    "        ax=ax3,\n",
    "        vmax=vmax,\n",
    "    )\n",
    "    ax3.set_title(\"The trained policy for the enemy (white)\")\n",
    "    fig.suptitle(f\"Headmap of where stones can be placed on turn {turn}\")\n",
    "    plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The analysis reveals that the chance of capturing a cornerstone is significantly higher when playing against a random player compared to playing against the trained policy. This is because the trained policy understands the importance of cornerstones and tends to capture them faster, while also avoiding giving the same opportunity to the opponent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to demonstrate this is by visualizing the mean chance of capturing one of the four cornerstones over time, as shown in the plot below. This provides a compact representation of the trend over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 292,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 75216 characters]\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "a = np.mean(_l_mean_actions_possible[:, [0, -1]][:, :, [0, -1]], axis=(1, 2))\n",
    "b = np.mean(_r_mean_actions_possible[:, [0, -1]][:, :, [0, -1]], axis=(1, 2))\n",
    "c = np.mean(mean_poss_turn[:, [0, -1]][:, :, [0, -1]], axis=(1, 2))\n",
    "\n",
    "df = pd.DataFrame(\n",
    "    [a, b, c],\n",
    "    index=[\"QL-Policy vs. Random\", \"Random vs. QL-Policy\", \"Random vs. Random\"],\n",
    ").T.plot(title=\"QL-Policy corner playable\", xlim=(0, 69))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Corner stone capture count\n",
    "When counting the number of corners captured, it's evident that the QL policy has some understanding of their importance. However, since the result is not absolutely dependent on those corners it can't be concluded that they should be overly prioritized, and an outcome around 3 would have been expected. The average of 2.7 corners captured at the end of the game, though not very high, shows a positive trend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 328,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QL-Policy vs. Random    2.748\n",
       "Random vs. QL-Policy    2.507\n",
       "Random vs. Random       1.952\n",
       "dtype: float64"
      ]
     },
     "execution_count": 328,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.sum(np.mean(_l_board_history[-1][:, [0, -1]][:, :, [0, -1]], axis=0))\n",
    "b = -np.sum(np.mean(_r_board_history[-1][:, [0, -1]][:, :, [0, -1]], axis=0))\n",
    "c = np.sum(np.mean(_board_history[-1][:, [0, -1]][:, :, [0, -1]], axis=0))\n",
    "\n",
    "(\n",
    "    pd.Series(\n",
    "        [a, b, c],\n",
    "        index=[\"QL-Policy vs. Random\", \"Random vs. QL-Policy\", \"Random vs. Random\"],\n",
    "    )\n",
    "    + 2\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Time till capture\n",
    "\n",
    "An additional aspect to analyze is the time it takes to capture a corner when it becomes available. As capturing a corner is critical in Othello strategy, it is expected that the AI would priorities its capture. However, since the opponent is usually not able to capture the same corner at the same moment, the AI may not need to hurry. To test this hypothesis, we measured the time it took for the AI to capture a corner and compared it to the time it took for the AI to capture a non-corner cell. The results showed that the AI captured corners faster, although not by as much as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 322,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "QL-Policy vs. Random    2.727500\n",
       "Random vs. QL-Policy    2.995250\n",
       "Random vs. Random       3.309563\n",
       "dtype: float64"
      ]
     },
     "execution_count": 322,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.Series(\n",
    "    {\n",
    "        \"QL-Policy vs. Random\": np.mean(\n",
    "            np.sum(_l_actions_possible[::2], axis=0)[:, [0, -1]][:, :, [0, -1]]\n",
    "        ),\n",
    "        \"Random vs. QL-Policy\": np.mean(\n",
    "            np.sum(_r_actions_possible[1::2], axis=0)[:, [0, -1]][:, :, [0, -1]]\n",
    "        ),\n",
    "        \"Random vs. Random\": np.mean(\n",
    "            (np.sum(_poss_turns, axis=0) / 2)[:, [0, -1]][:, :, [0, -1]]\n",
    "        ),\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "With the corner capture behavior of this policy I was quit happy even if it was not as extreme as I expected. It goes into the right direction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Analyzing Symmetry in the ANN's Behavior\n",
    "\n",
    "There are several ways to assess symmetry, but I used the R2 metric, which is sometimes referred to as the normed error in Deep Learning. It's important to note that this is not a loss function, but rather a way to measure how symmetric the ANN's behavior is.\n",
    "\n",
    "The R2 score is computed as follows: $R_2 = 1 - \\frac{MSE}{VAR}$ In my analysis, I calculated the error by flipping an axis, generating a policy, and then reversing the flip to its directly calculated opposite. The primary advantage of this formula is that it normalizes the error.\n",
    "\n",
    "A score of 1 indicates that there is no error, while a score of 0 suggests that there is only random noise. Negative values indicate that the connection is worse than random."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The VAR value for the R2 value calucates as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "_board_history, _action_history = load_board_history()\n",
    "_board_history.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "base_policy_results = example_policy._internal_policy(\n",
    "    _board_history.reshape((-1, 8, 8))\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.011764458"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var = np.var(base_policy_results)\n",
    "var"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "When flipping the first axis the following is returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9503335431218147"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    1\n",
    "    - np.var(\n",
    "        base_policy_results\n",
    "        - example_policy._internal_policy(\n",
    "            _board_history.reshape((-1, 8, 8))[:, ::-1, :]\n",
    "        )[:, ::-1, :]\n",
    "    )\n",
    "    / var\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When flipping the second axis the score reached is very similar which supports that the symetry of the game was learned somewhat OK."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9560678526759148"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(\n",
    "    1\n",
    "    - np.var(\n",
    "        base_policy_results\n",
    "        - example_policy._internal_policy(\n",
    "            _board_history.reshape((-1, 8, 8))[:, :, ::-1]\n",
    "        )[:, :, ::-1]\n",
    "    )\n",
    "    / var\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "Since discount factors make out a huge part of the Q-Value this should be compensated for by this analysis.Those discount values can be found quit clearly in the Q-Valuenvariances below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2.69594672e-03, 1.61979976e-03, 3.16265994e-03, 2.86013912e-03,\n",
       "       3.88444145e-03, 4.10827110e-03, 4.55618650e-03, 5.41831693e-03,\n",
       "       5.22514898e-03, 6.72813132e-03, 5.98392496e-03, 8.03542975e-03,\n",
       "       6.71476265e-03, 9.31352191e-03, 7.40469201e-03, 1.04836905e-02,\n",
       "       8.15843511e-03, 1.17105739e-02, 8.97589605e-03, 1.29910205e-02,\n",
       "       9.83012654e-03, 1.42743532e-02, 1.07203163e-02, 1.54738314e-02,\n",
       "       1.15436241e-02, 1.66677851e-02, 1.24594867e-02, 1.80033017e-02,\n",
       "       1.33895278e-02, 1.92428790e-02, 1.42273121e-02, 2.02678796e-02,\n",
       "       1.50251873e-02, 2.12085303e-02, 1.56881604e-02, 2.20195204e-02,\n",
       "       1.63181052e-02, 2.27781329e-02, 1.68893356e-02, 2.33910922e-02,\n",
       "       1.72991194e-02, 2.36464869e-02, 1.75159350e-02, 2.37203576e-02,\n",
       "       1.75640993e-02, 2.35915985e-02, 1.72887910e-02, 2.27049645e-02,\n",
       "       1.66701656e-02, 2.14512497e-02, 1.55042894e-02, 1.96573995e-02,\n",
       "       1.40535031e-02, 1.72253978e-02, 1.20143285e-02, 1.38071319e-02,\n",
       "       9.17551387e-03, 9.47508123e-03, 5.56008052e-03, 3.95872677e-03,\n",
       "       1.55381148e-03, 2.98189756e-04, 1.54811903e-04, 5.28804194e-05,\n",
       "       4.63396464e-05, 4.13985799e-05, 4.09298846e-05, 3.93082191e-05,\n",
       "       3.93082191e-05, 3.93082191e-05], dtype=float32)"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var_sqt = np.var(base_policy_results.reshape(70, -1, 8, 8), axis=(1, 2, 3))\n",
    "var_sqt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.1676074e-05, 6.2680389e-05, 1.2044878e-04, 1.1384000e-04,\n",
       "       1.7748313e-04, 1.8506356e-04, 2.5800613e-04, 2.6801604e-04,\n",
       "       3.4888665e-04, 3.6445708e-04, 4.3677850e-04, 4.4575133e-04,\n",
       "       5.1948725e-04, 5.3083879e-04, 6.0994463e-04, 6.1461091e-04,\n",
       "       7.0802838e-04, 7.0421951e-04, 7.8046869e-04, 7.7499606e-04,\n",
       "       8.6105539e-04, 8.5808098e-04, 9.4744092e-04, 9.2485105e-04,\n",
       "       1.0099772e-03, 9.5811568e-04, 1.0360932e-03, 9.9020952e-04,\n",
       "       1.0699228e-03, 1.0346725e-03, 1.0979769e-03, 1.0489059e-03,\n",
       "       1.1245861e-03, 1.0655295e-03, 1.1300436e-03, 1.0458149e-03,\n",
       "       1.1143789e-03, 1.0437473e-03, 1.1256501e-03, 1.0107887e-03,\n",
       "       1.0733304e-03, 9.7007811e-04, 1.0121627e-03, 8.9826877e-04,\n",
       "       9.4787223e-04, 8.2494231e-04, 8.6397259e-04, 7.4023078e-04,\n",
       "       7.6400751e-04, 6.4820121e-04, 6.5807183e-04, 5.4538535e-04,\n",
       "       5.4212776e-04, 4.3356448e-04, 4.1144638e-04, 3.0260321e-04,\n",
       "       2.7869738e-04, 1.8341088e-04, 1.4314732e-04, 6.4388187e-05,\n",
       "       2.5075868e-05, 4.3099076e-06, 1.3687046e-06, 6.8730822e-07,\n",
       "       6.5397217e-07, 6.2672098e-07, 6.2625560e-07, 6.2597036e-07,\n",
       "       6.2597036e-07, 6.2597047e-07], dtype=float32)"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "var_mse = np.array(\n",
    "    [\n",
    "        np.var(\n",
    "            base_policy_results.reshape(70, -1, 8, 8)[i]\n",
    "            - example_policy._internal_policy(_board_history[i, :, ::-1, :])[:, ::-1, :]\n",
    "        )\n",
    "        for i in range(70)\n",
    "    ]\n",
    ")\n",
    "var_mse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.9882505 , 0.9613036 , 0.9619154 , 0.96019775, 0.9543092 ,\n",
       "       0.95495343, 0.94337237, 0.9505352 , 0.9332293 , 0.9458309 ,\n",
       "       0.92700803, 0.94452673, 0.9226351 , 0.9430034 , 0.9176273 ,\n",
       "       0.94137454, 0.91321516, 0.93986464, 0.9130484 , 0.94034374,\n",
       "       0.9124065 , 0.9398865 , 0.9116219 , 0.94023126, 0.9125078 ,\n",
       "       0.9425169 , 0.91684306, 0.94499844, 0.9200926 , 0.9462309 ,\n",
       "       0.9228261 , 0.94824785, 0.92515326, 0.94975936, 0.9279684 ,\n",
       "       0.9525051 , 0.93170905, 0.9541777 , 0.9333514 , 0.95678747,\n",
       "       0.9379546 , 0.9589758 , 0.9422147 , 0.9621309 , 0.94603354,\n",
       "       0.96503234, 0.950027  , 0.96739787, 0.95416915, 0.9697826 ,\n",
       "       0.9575555 , 0.97225547, 0.961424  , 0.9748299 , 0.9657537 ,\n",
       "       0.97808355, 0.96962595, 0.9806428 , 0.9742544 , 0.98373514,\n",
       "       0.9838617 , 0.9855464 , 0.9911589 , 0.9870026 , 0.9858874 ,\n",
       "       0.9848613 , 0.9846993 , 0.9840753 , 0.9840753 , 0.9840753 ],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "1 - var_mse / var_sqt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.95247847"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(1 - var_mse / var_sqt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scores calculated above indicate that the policies exhibit almost symmetrical behavior across the two validated axes. Although the same has not been validated for the third axis, it is assumed to be similar. This demonstrates that the learning behavior is progressing in the right direction, even if the policy's score is lower than expected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "The implementation of Q-learning using artificial neural networks (ANNs) for the board game Othello proved to be a challenging task. Although I had planned to implement increasingly complex policies, the amount of work required for a reinforcement learning (RL) project, including Q-learning, was much larger than expected. Nonetheless, I am satisfied with the progress made, considering the limited amount of time available. Although the networks did not clearly converge, they showed promise, and I am uncertain how to define the best values for parameters such as $\\gamma$, $\\epsilon$, who_won_fraction, and final_score_fraction.\n",
    "\n",
    "Bug chasing was a significant challenge during the implementation, leading to multiple restarts of the training process and limiting the amount of computation. The use of memory replay and Q-learning, which takes the discounted maximum reward for the next state as a reward, would have been useful for overcoming this issue, but I did not explore this approach. The next step would have been to implement policy gradient and connect it to Q-learning to build an actor-critic, which would have improved the stability and reduced the need for validating as many actions as possible during training.\n",
    "\n",
    "This version of Q-learning using ANNs is more of a hashed version of Q-learning via deep Q-networks (DQN), which defines its loss as $L_{DQN} = r + \\gamma \\max_{a'} Q(s', a'; \\theta^-_i) - Q(s, a; \\theta_i)$. However, we worked with discounted values and not a discounted backpropagation through the ANN, allowing for exploration without skewing the rewards. Nonetheless, this approach needs to be balanced with other values that tend to skew the results towards the mean and may lead to mediocre results.\n",
    "\n",
    "In retrospect, I underestimated the effort required for this project. The pre-work and simulation of the game were more complex than expected, and working without proper testing and only using asserts added to the difficulties. Using Jupyter notebooks for such a large codebase was also not ideal. Although adding tests helped, the implementation of custom reward functions, with changes in perspective, axis, and other values, was still challenging to code and debug. Next time, I will start with testing at an early stage, even if it means creating test data, to speed up development.\n",
    "\n",
    "Although the ANN structure seems to work, it is limited by its environment. One possible change would be to eliminate the need for discounts, as Othello has a maximum of 60 turns. As such, it can be argued that no discount is necessary since it is possible to avoid the end of the game or an endless loop."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sources\n",
    "\n",
    "* Game rules and example board images [https://en.wikipedia.org/wiki/Reversi](https://en.wikipedia.org/wiki/Reversi)\n",
    "* Game rules and example game images [https://de.wikipedia.org/wiki/Othello_(Spiel)](https://de.wikipedia.org/wiki/Othello_(Spiel))\n",
    "* Game strategy examples [https://de.wikipedia.org/wiki/Computer-Othello](https://de.wikipedia.org/wiki/Computer-Othello)\n",
    "* Image for 8 directions [https://www.researchgate.net/journal/EURASIP-Journal-on-Image-and-Video-Processing-1687-5281](https://www.researchgate.net/journal/EURASIP-Journal-on-Image-and-Video-Processing-1687-5281)\n",
    "* Deepl Learning with PyTorch 1.x (ISBN 978-1-83855-300-5)\n",
    "\n",
    "ChatGPG was used to refactor some code snippets and as an advanced spell checker."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Memory usage\n",
    "\n",
    "The jupyter notebook uses a lot of memory. The cell below monitors the statically allocated memory to get some kind of feeling where memory is completely wasted.\n",
    "\n",
    "The code snipped was copied and only slightly modified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "\n",
    "def sizeof_fmt(num, suffix=\"B\"):\n",
    "    \"\"\"by Fred Cirera,  https://stackoverflow.com/a/1094933/1870254, modified\"\"\"\n",
    "    for unit in [\"\", \"Ki\", \"Mi\", \"Gi\", \"Ti\", \"Pi\", \"Ei\", \"Zi\"]:\n",
    "        if abs(num) < 1024.0:\n",
    "            return \"%3.1f %s%s\" % (num, unit, suffix)\n",
    "        num /= 1024.0\n",
    "    return \"%.1f %s%s\" % (num, \"Yi\", suffix)\n",
    "\n",
    "\n",
    "for name, size in sorted(\n",
    "    ((name, sys.getsizeof(value)) for name, value in list(locals().items())),\n",
    "    key=lambda x: -x[1],\n",
    ")[:20]:\n",
    "    print(\"{:>30}: {:>8}\".format(name, sizeof_fmt(size)))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  },
  "toc-autonumbering": true,
  "toc-showcode": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
