03. Analyze.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Analyze Data\n",
    "\n",
    "[![Python Data Science](https://apmonitor.com/che263/uploads/Begin_Python/DataScience03.png)](https://www.youtube.com/watch?v=5yv_ID4YNTI&list=PLLBUgWXdTBDg1Qgmwt4jKtVn9BWh5-zgy \"Python Data Science\")\n",
    "\n",
    "Once data is read into Python, a first step is to analyze the data with summary statistics. This is especially true if the data set is large. Summary statistics include the count, mean, standard deviation, maximum, minimum, and quartile information for the data columns. \n",
    "\n",
    "![idea](https://apmonitor.com/che263/uploads/Begin_Python/idea.png)\n",
    "\n",
    "### Generate Data\n",
    "\n",
    "Run the next cell to:\n",
    "\n",
    "- Generate `n` linearly spaced values betweeen `0` and `n-1` with `np.linspace(start,end,count)`\n",
    "- Draw random samples from a uniform distribution between 0 and 1 with `np.random.rand(count)`\n",
    "- Draw random samples from a normal (Gaussian) distribution with `np.random.normal(mean,std,count)`\n",
    "- Combine `time`, `x`, and `y` with a vertical stack `np.vstack` and transpose `.T` for column oriented data.\n",
    "- Save CSV text file `03-data.csv` with header `time,x,y`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.random.seed(0)\n",
    "n = 1000\n",
    "time = np.linspace(0,n-1,n)\n",
    "x = np.random.rand(n)\n",
    "y = np.random.normal(1,1,n)\n",
    "data = np.vstack((time,x,y)).T\n",
    "np.savetxt('03-data.csv',data,header='time,x,y',delimiter=',',comments='')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![idea](https://apmonitor.com/che263/uploads/Begin_Python/idea.png)\n",
    "\n",
    "### Display Data Distributions\n",
    "\n",
    "The histogram is a preview of how to create graphics so that data can be evaluated visually. [04. Visualize](https://github.com/APMonitor/data_science/blob/master/04.%20Visualize.ipynb) shows how to create plots to analyze data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.hist(x,10,label='x')\n",
    "plt.hist(y,60,label='y',alpha=0.7)\n",
    "plt.ylabel('Count'); plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![idea](https://apmonitor.com/che263/uploads/Begin_Python/idea.png)\n",
    "\n",
    "### Data Analysis with `numpy`\n",
    "\n",
    "The `np.loadtxt` function reads the CSV data file `03-data.csv`. Numpy calculates `size` (dimensions), `mean` (average), `std` (standard deviation), and `median` as summary statistics. If you don't specify the `axis` then `numpy` gives a statistic across both the rows (`axis=0`) and columns (`axis=1`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "data = np.loadtxt('03-data.csv',delimiter=',',skiprows=1)\n",
    "\n",
    "print('Dimension (rows,columns):')\n",
    "print(np.size(data,0),np.size(data,1))\n",
    "\n",
    "print('Average:')\n",
    "print(np.mean(data,axis=0))\n",
    "\n",
    "print('Standard Deviation:')\n",
    "print(np.std(data,0))\n",
    "\n",
    "print('Median:')\n",
    "print(np.median(data,0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![expert](https://apmonitor.com/che263/uploads/Begin_Python/expert.png)\n",
    "\n",
    "### Analyze data\n",
    "\n",
    "1. Calculate the mean, standard deviation, and median of `x*y`\n",
    "2. Calculate the `skew` of `x*y` with the `scipy.stats` [skew function](https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.skew.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![idea](https://apmonitor.com/che263/uploads/Begin_Python/idea.png)\n",
    "\n",
    "### Data Analysis with `pandas`\n",
    "\n",
    "Pandas simplifies the data analysis with the `.describe()` function that is a method of `DataFrame` that is created with `pd.read_csv()`. Note that the data file can either be a local file name or a web-address such as \n",
    "\n",
    "```python\n",
    "url='https://apmonitor.com/pdc/uploads/Main/tclab_data2.txt'\n",
    "data = pd.read_csv(url)\n",
    "data.describe()\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "data = pd.read_csv('03-data.csv')\n",
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![expert](https://apmonitor.com/che263/uploads/Begin_Python/expert.png)\n",
    "\n",
    "### TCLab Activity\n",
    "\n",
    "![connections](https://apmonitor.com/che263/uploads/Begin_Python/connections.png)\n",
    "\n",
    "### Generate Data Set 1 \n",
    "\n",
    "Generate a file from the TCLab data with seconds (`t`), heater levels (`Q1` and `Q2`), and temperatures (`lab.T1` and `lab.T2`). Record data every second for 120 seconds and change the heater levels every 30 seconds to a random number between 0 and 80 with `np.random.randint()`. There is no need to change this program, only run it for 2 minutes to collect the data. If you do not have a TCLab device, read a data file 1 from [an online link](https://apmonitor.com/do/uploads/Main/tclab_dyn_data2.txt)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tclab, time, csv\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "try:\n",
    "    # connect to TCLab if available\n",
    "    n = 120 \n",
    "    with open('03-tclab1.csv',mode='w',newline='') as f:\n",
    "        cw = csv.writer(f)\n",
    "        cw.writerow(['Time','Q1','Q2','T1','T2'])\n",
    "        with tclab.TCLab() as lab:\n",
    "            print('t Q1 Q2 T1    T2')\n",
    "            for t in range(n):\n",
    "                if t%30==0:\n",
    "                    Q1 = np.random.randint(0,81)\n",
    "                    Q2 = np.random.randint(0,81)\n",
    "                    lab.Q1(Q1); lab.Q2(Q2)\n",
    "                cw.writerow([t,Q1,Q2,lab.T1,lab.T2])\n",
    "                if t%5==0:\n",
    "                    print(t,Q1,Q2,lab.T1,lab.T2)\n",
    "                time.sleep(1)\n",
    "    file = '03-tclab1.csv'\n",
    "    data1=pd.read_csv(file)\n",
    "except:\n",
    "    print('No TCLab device found, reading online file')\n",
    "    url = 'http://apmonitor.com/do/uploads/Main/tclab_dyn_data2.txt'\n",
    "    data1=pd.read_csv(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read Data Set 2\n",
    "\n",
    "Use `requests` to download a sample TCLab data file for the analysis. It is saved as `03-tclab2.csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import os\n",
    "url = 'http://apmonitor.com/pdc/uploads/Main/tclab_data2.txt'\n",
    "r = requests.get(url)\n",
    "with open('03-tclab2.csv', 'wb') as f:\n",
    "    f.write(r.content)\n",
    "    \n",
    "print('File 03-tclab2.csv retrieved to current working directory: ')\n",
    "print(os.getcwd())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Analysis\n",
    "\n",
    "Read the files `03-tclab1.csv` and `03-tclab2.csv` and display summary statistics for each with `data.describe()`. Use the summary statistics to compare the number of samples and differences in average and standard deviation value for `T1` and `T2`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}