{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# fecon235: Introductory documentation\n",
    "\n",
    "Here we discuss the usage of the *fecon235* repository \n",
    "for the casual user (while the development of the API \n",
    "is still in progress). \n",
    "\n",
    "**As of v4, all *modules* will work with Python 2.7 and 3 series.** \n",
    "However, some of the pre-2016 *notebooks* may still have python2 \n",
    "idioms and Linux dependencies. Those are easy to fix as we update. \n",
    "Our goal is cross-platform performance (Linux, Mac, and Windows) \n",
    "as well as compliance with both Python kernels available to \n",
    "Jupyter notebooks (forked from IPython).\n",
    "\n",
    "To see examples of code, please pick out a subject of interest under \n",
    "the **nb** directory, and view that notebook at GitHub. \n",
    "Better yet, fork this project, and execute the notebook locally as \n",
    "you interactively experiment. \n",
    "For developers, the main modules are located under the **lib** directory.\n",
    "\n",
    "## Importing the project\n",
    "\n",
    "For python3 conformity, we have adopted absolute_import \n",
    "throughout this project. \n",
    "So first be sure that your PYTHONPATH can lead up to \n",
    "the fecon235 directory. Then the following import \n",
    "permits *easy command access*. The top-level module is \n",
    "customarily given the same name as the project. In our case, \n",
    "it conveniently unifies and exposes our essential lib modules \n",
    "(older notebooks imported yi-prefixed modules individually)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#  Call the MAIN module: \n",
    "from fecon235.fecon235 import *\n",
    "#  This loose import style is acceptable only within \n",
    "#  interactive environments outside of any fecon235 packages.\n",
    "#  (Presence of __init__.py in a directory indicates \n",
    "#  it is a \"package.\") \n",
    "#\n",
    "#  These directories: nb and tests, are explicitly NOT packages."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use fecon235 in other projects, here are some proper examples:\n",
    "\n",
    "    from fecon235 import fecon235 as fe\n",
    "    from fecon235.lib import yi_secform\n",
    "    \n",
    "If we had used the first example in our notebooks, \n",
    "a function would require extra typing, \n",
    "e.g. *fe.get()* instead of plain *get()*. \n",
    "Any lib module can be imported directly \n",
    "if specialized procedures are required. \n",
    "The second example is used to parse SEC forms. \n",
    "An inventory of available procedures is \n",
    "provided below as Appendix 1.\n",
    "\n",
    "### Every notebook states its dependencies and changes:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Dependencies:*\n",
    "\n",
    "- fecon235 repository https://github.com/rsvp/fecon235\n",
    "- Python: matplotlib, numpy, pandas\n",
    "     \n",
    "*CHANGE LOG*\n",
    "\n",
    "    2015-12-30  Add inventory of lib procedures as Appendix 1.\n",
    "    2015-12-28  First version of README notebook in docs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " ::  Python 2.7.10\n",
      " ::  IPython 4.0.0\n",
      " ::  jupyter 1.0.0\n",
      " ::  notebook 4.0.6\n",
      " ::  matplotlib 1.4.3\n",
      " ::  numpy 1.10.1\n",
      " ::  pandas 0.17.1\n",
      " ::  pandas_datareader 0.2.0\n",
      " ::  Repository: fecon235 v3.15.1216 develop\n",
      " ::  Timestamp: 2015-12-30, 19:22:33 UTC\n",
      " ::  $pwd: /media/yaya/virt15h/virt/dbx/Dropbox/ipy/fecon235/docs\n"
     ]
    }
   ],
   "source": [
    "#  PREAMBLE-p6.15.1223 :: Settings and system details\n",
    "from __future__ import absolute_import, print_function\n",
    "system.specs()\n",
    "pwd = system.getpwd()   # present working directory as variable.\n",
    "print(\" ::  $pwd:\", pwd)\n",
    "#  If a module is modified, automatically reload it:\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "#       Use 0 to disable this feature.\n",
    "\n",
    "#  Notebook DISPLAY options:\n",
    "#      Represent pandas DataFrames as text; not HTML representation:\n",
    "import pandas as pd\n",
    "pd.set_option( 'display.notebook_repr_html', False )\n",
    "#  Beware, for MATH display, use %%latex, NOT the following:\n",
    "#                   from IPython.display import Math\n",
    "#                   from IPython.display import Latex\n",
    "from IPython.display import HTML # useful for snippets\n",
    "#  e.g. HTML('<iframe src=http://en.mobile.wikipedia.org/?useformat=mobile width=700 height=350></iframe>')\n",
    "from IPython.display import Image \n",
    "#  e.g. Image(filename='holt-winters-equations.png', embed=True) # url= also works\n",
    "from IPython.display import YouTubeVideo\n",
    "#  e.g. YouTubeVideo('1j_HxD4iLn8', start='43', width=600, height=400)\n",
    "from IPython.core import page\n",
    "get_ipython().set_hook('show_in_pager', page.as_hook(page.display_page), 0)\n",
    "#  Or equivalently in config file: \"InteractiveShell.display_page = True\", \n",
    "#  which will display results in secondary notebook pager frame in a cell.\n",
    "\n",
    "#  Generate PLOTS inside notebook, \"inline\" generates static png:\n",
    "%matplotlib inline   \n",
    "#          \"notebook\" argument allows interactive zoom and resize."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Preamble for settings\n",
    "\n",
    "The preamble contains the latest shortcuts for notebook commands, \n",
    "but more importantly, it lists the specific dependencies \n",
    "which makes research **reproducible**. The \"Repository:\" line \n",
    "should indicate the annotated tag associated with the last good state \n",
    "of repository at the time of execution. The branch is then stated, \n",
    "which completes the analogue of *requirements.txt* for notebooks.\n",
    "\n",
    "The \"Timestamp:\" will indicate the staleness of the data. \n",
    "Notebooks have executed properly at the indicated time, \n",
    "and when committed to the repository. \n",
    "If notebooks are re-executed the most current data \n",
    "will be intentionally downloaded. \n",
    "Thus many observations in notebooks include their date. \n",
    "Changes upstream, in the meantime, can possibly generate  \n",
    "errors in re-executed fecon235 notebooks \n",
    "(esp. deprecated pandas functions).\n",
    "\n",
    "Notebooks implicitly function as integration tests \n",
    "of the underlying code, and thus reveal technical failures. \n",
    "Another notebook will cover unit tests in the *tests* directory \n",
    "for developers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Internal queries and documentation\n",
    "\n",
    "Notebooks have a wonderful feature: **?** and **??** \n",
    "which give further information on variables, functions, \n",
    "classes, etc. And where to exactly look for the source.\n",
    "\n",
    "The second question mark gives more verbose answers. \n",
    "All our codes have detailed docstrings and comments, \n",
    "so we strive to be self-documenting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[1;31mType:        \u001b[0mmodule\n",
       "\u001b[1;31mString form: \u001b[0m<module 'fecon235.lib.yi_0sys' from '/home/yaya/Dropbox/ipy/fecon235/lib/yi_0sys.pyc'>\n",
       "\u001b[1;31mFile:        \u001b[0m~/Dropbox/ipy/fecon235/lib/yi_0sys.py\n",
       "\u001b[1;31mDocstring:\u001b[0m\n",
       "_______________|  yi_0sys.py : system and date functions including specs.\n",
       "\n",
       "Code in this module must be compatible with both Python 2 and 3.\n",
       "It is a bridge and a guardian between the two Pythons.\n",
       "\n",
       "For example, it is used in the preamble of fecon235 Jupyter notebooks.\n",
       "\n",
       "\n",
       "REFERENCES:\n",
       "- Compatible IDIOMS: http://python-future.org/compatible_idioms.html\n",
       "                     Nice presentation.\n",
       "\n",
       "- SIX module is exhaustive: https://pythonhosted.org/six/\n",
       "        Single file source: https://bitbucket.org/gutworth/six\n",
       "\n",
       "\n",
       "CHANGE LOG  For latest version, see https://github.com/rsvp/fecon235\n",
       "2015-12-29  For errf in gitinfo(), our dev_null instead of os.devnull\n",
       "               Add minimumPandas variable esp. for tests.\n",
       "2015-12-27  Get jupyter version among specs().\n",
       "               Fix run command to accept errf argument.\n",
       "               For specs(), move gitinfo try clause therein.\n",
       "2015-12-23  Add run command and gitinfo functions.\n",
       "               Update to PREAMBLE-p6.15.1223\n",
       "2015-12-19  python3 compatible: absolute_import\n",
       "2015-12-03  First version."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#  What the heck is \"system\" mentioned in the preamble?\n",
    "system?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting data\n",
    "\n",
    "Our project currently has free access to data on equities, \n",
    "government bonds, commodities, and futures -- as well as, \n",
    "a full range of economic statistics. The key is finding \n",
    "the string which will retrieve the desired time-series. \n",
    "(A detailed *docs* notebook dedicated to data retrieval is forthcoming.)\n",
    "\n",
    "### Sample: Unemployment rate\n",
    "\n",
    "Let's go through an example. The function **get** is \n",
    "designed as an overlord over specialized get functions. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#  Assign a name to a dataframe\n",
    "#  that will contain monthly unemployment rates.\n",
    "\n",
    "unem = get( m4unemp )\n",
    "#           m4 implies monthly frequency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[1;31mType:        \u001b[0mstr\n",
       "\u001b[1;31mString form: \u001b[0mUNRATE\n",
       "\u001b[1;31mLength:      \u001b[0m6\n",
       "\u001b[1;31mDocstring:\u001b[0m\n",
       "str(object='') -> string\n",
       "\n",
       "Return a nice string representation of the object.\n",
       "If the argument is a string, the return value is the same object."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#  But does m4unemp really represent?\n",
    "m4unemp?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Variables for data\n",
    "\n",
    "So we see that m4unemp is our variable holding a string \"UNRATE\". \n",
    "That string is the internal code used by FRED, the database \n",
    "at the Federal Reserve Bank in St. Louis. Our variables are \n",
    "generally easier to remember, and mentions the frequency. \n",
    "\n",
    "If there is no special variable, one can \n",
    "always get(\"string\") to directly retrieve data.\n",
    "\n",
    "Sometimes a variable for a data set may trigger a \n",
    "subroutine which post-processes the original data \n",
    "(e.g. see our inflation measures), or brings offline \n",
    "data into memory (for example, our compressed CSV files may \n",
    "contain synthetic data, e.g. the euro exchange rate \n",
    "years prior to its official circulation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#  Illustrate slicing: 1997 <= unem <= 2007:\n",
    "unem07 = unem['1997':'2007']\n",
    "#  Verify below by Head and Tail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                Y\n",
      "count  132.000000\n",
      "mean     4.908333\n",
      "std      0.646073\n",
      "min      3.800000\n",
      "25%      4.400000\n",
      "50%      4.700000\n",
      "75%      5.500000\n",
      "max      6.300000\n"
     ]
    }
   ],
   "source": [
    "#  Quick summary:\n",
    "stat( unem07 )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                Y\n",
      "count  132.000000\n",
      "mean     4.908333\n",
      "std      0.646073\n",
      "min      3.800000\n",
      "25%      4.400000\n",
      "50%      4.700000\n",
      "75%      5.500000\n",
      "max      6.300000\n",
      "\n",
      " ::  Index on min:\n",
      "Y   2000-04-01\n",
      "dtype: datetime64[ns]\n",
      "\n",
      " ::  Index on max:\n",
      "Y   2003-06-01\n",
      "dtype: datetime64[ns]\n",
      "\n",
      " ::  Head:\n",
      "              Y\n",
      "T              \n",
      "1997-01-01  5.3\n",
      "1997-02-01  5.2\n",
      "1997-03-01  5.2\n",
      "1997-04-01  5.1\n",
      "1997-05-01  4.9\n",
      "1997-06-01  5.0\n",
      "1997-07-01  4.9\n",
      "\n",
      " ::  Tail:\n",
      "              Y\n",
      "T              \n",
      "2007-06-01  4.6\n",
      "2007-07-01  4.7\n",
      "2007-08-01  4.6\n",
      "2007-09-01  4.7\n",
      "2007-10-01  4.7\n",
      "2007-11-01  4.7\n",
      "2007-12-01  5.0\n",
      "\n",
      " ::  Correlation matrix:\n",
      "   Y\n",
      "Y  1\n"
     ]
    }
   ],
   "source": [
    "#  More verbose statistical summary:\n",
    "stats( unem07 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The correlation matrix has only one entry above. \n",
    "This is because *stats()* is designed to take \n",
    "a dataframe with multiple columns as argument.\n",
    "Let's see how the function is written \n",
    "and where we can find it in the filesystem.\n",
    "\n",
    "Indeed *stats()* calls our *cormatrix()* to compute \n",
    "the correlation matrix. And one can go on \n",
    "further to query that function... eventually \n",
    "that query could reach a core numerical package \n",
    "such as numpy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[1;31mSignature: \u001b[0m\u001b[0mstats\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mdataframe\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
       "\u001b[1;31mSource:\u001b[0m\n",
       "def stats( dataframe ):\n",
       "     '''VERBOSE statistics on given dataframe; CORRELATIONS without regression.'''\n",
       "     print(dataframe.describe())\n",
       "     print()\n",
       "     print(\" ::  Index on min:\")\n",
       "     print(dataframe.idxmin())\n",
       "     print()\n",
       "     print(\" ::  Index on max:\")\n",
       "     print(dataframe.idxmax())\n",
       "     print()\n",
       "     print(\" ::  Head:\")\n",
       "     print(head( dataframe ))\n",
       "     print()\n",
       "     print(\" ::  Tail:\")\n",
       "     print(tail( dataframe ))\n",
       "     print()\n",
       "     print(\" ::  Correlation matrix:\")\n",
       "     print(cormatrix( dataframe ))\n",
       "     return\n",
       "\u001b[1;31mFile:      \u001b[0m~/Dropbox/ipy/fecon235/lib/yi_1tools.py\n",
       "\u001b[1;31mType:      \u001b[0mfunction"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "stats??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# #  Uncomment to see how numpy computes something simple as absolute value:\n",
    "# np.abs??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## Computing from the data\n",
    "\n",
    "The analysis of data is at the heart of this project. \n",
    "Specific computational tools will be covered in \n",
    "other notebooks under the *docs* directory.\n",
    "\n",
    "To follow up on unemployment example, see https://git.io/fed \n",
    "which scores the Federal Reserve on their dual mandate. \n",
    "Visualization is provided by our plot tools, \n",
    "which as a by-product discredits the Phillips curve \n",
    "as adequate causal theory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Questions or bugs\n",
    "\n",
    "- Chat with fellow users at Gitter: https://gitter.im/rsvp/fecon235\n",
    "\n",
    "- Report an issue at https://github.com/rsvp/fecon235/issues\n",
    "\n",
    "- Summarize your usage solution at our wiki: https://github.com/rsvp/fecon235/wiki\n",
    "\n",
    "- Blame the lead developer: *Adriano* [rsvp.github.com](https://rsvp.github.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Appendix 1: Procedures defined in lib modules\n",
    "\n",
    "As of 2015-12-30, many of these procedures and functions \n",
    "are unified by the top level module **fecon235.py** \n",
    "which also simplifies their usage, for example, \n",
    "get() and plot():\n",
    "\n",
    "#### yi_0sys.py\n",
    "\n",
    "     getpwd():\n",
    "         Get present working directory (Linux command is pwd).\n",
    "     program():\n",
    "         Get name of present script; works cross-platform.\n",
    "     warn( message, stub=\"WARNING:\", prefix=\" !. \"):\n",
    "         Write warning solely to standard error.\n",
    "     die( message, errcode=1, prefix=\" !! \"):\n",
    "         Gracefully KILL script, optionally specifying error code.\n",
    "     date( hour=True, utc=True, localstr=' Local' ):\n",
    "         Get date, and optionally time, as ISO string representation.\n",
    "     pythontup():\n",
    "         Represent invoked Python version as an integer 3-tuple.\n",
    "     versionstr( module=\"IPython\" ):\n",
    "         Represent version as a string, or None if not installed.\n",
    "     versiontup( module=\"IPython\" ):\n",
    "         Parse version string into some integer 3-tuple.\n",
    "     version( module=\"IPython\" ):\n",
    "         Pretty print Python or module version info.\n",
    "     utf( immigrant, xnl=True ):\n",
    "         Convert to utf-8, and possibly delete new line character.\n",
    "     run( command, xnl=True, errf=None ):\n",
    "         RUN **quote and space insensitive** SYSTEM-LEVEL command.\n",
    "     gitinfo():\n",
    "         From git, get repo name, current branch and annotated tag.\n",
    "     specs():\n",
    "         Show ecosystem specifications, including execution timestamp.\n",
    "     ROSETTA STONE FUNCTIONS approximately bridging Python 2 and 3.\n",
    "     endmodule():\n",
    "         Procedure after __main__ conditional in modules.\n",
    "\n",
    "#### yi_1tools.py\n",
    "\n",
    "     nona( df ):\n",
    "          Eliminate any row in a dataframe containing NA, NaN nulls.\n",
    "     head( dfx, n=7 ):\n",
    "          Quick look at the INITIAL data point(s).\n",
    "     tail( dfx, n=7 ):\n",
    "          Quick look at the LATEST data point(s).\n",
    "     tailvalue( df, pos=0, row=1 ):\n",
    "          Seek (last) row of dataframe, then the element at position pos.\n",
    "     div( numerator, denominator, floor=False ):\n",
    "          Division via numpy for pandas, Python 2 and 3 compatibility.\n",
    "     dif( dfx, freq=1 ):\n",
    "          Lagged difference for pandas series.\n",
    "     pcent( dfx, freq=1 ):\n",
    "          PERCENTAGE CHANGE method for pandas.\n",
    "     georet( dfx, yearly=256 ):\n",
    "          Compute geometric mean return in a summary list.\n",
    "     zeroprice( rate, duration=9, yearly=2, face=100 ):\n",
    "          Compute price of zero-coupon bond given its duration.\n",
    "     ema( y, alpha=0.20 ):\n",
    "          EXPONENTIAL MOVING AVERAGE using traditional weight arg.\n",
    "     normalize( dfy ):\n",
    "          Center around mean zero and standardize deviation.\n",
    "     correlate( dfy, dfx, type='pearson' ):\n",
    "          CORRELATION FUNCTION between series using pandas method.\n",
    "     cormatrix( dataframe, type='pearson' ):\n",
    "          PAIRWISE CORRELATIONS within a dataframe using pandas method.\n",
    "     regressformula( df, formula ):\n",
    "          Helper function for statsmodel linear regression using formula.\n",
    "     regressTIME( dfy, col='Y' ):\n",
    "          Regression on time since such index cannot be an independent variable.\n",
    "     regresstime( dfy, col='Y' ):\n",
    "          Regression on time since such index cannot be an independent variable.\n",
    "     regresstimeforecast( dfy, h=24, col='Y' ):\n",
    "          Forecast h-periods ahead based on linear regression on time.\n",
    "     detrend( dfy, col='Y' ):\n",
    "          Detread using linear regression on time.\n",
    "     detrendpc( dfy, col='Y' ):\n",
    "          Detread using linear regression on time; percent deviation.\n",
    "     detrendnorm( dfy, col='Y' ):\n",
    "          Detread using linear regression on time, then normalize.\n",
    "     regress( dfy, dfx ):\n",
    "         Perform LINEAR REGRESSION, a.k.a. Ordinary Least Squares.\n",
    "     stat2( dfy, dfx ):\n",
    "          Quick STATISTICAL SUMMARY and regression on two variables\n",
    "     stat( dataframe, pctiles=[0.25, 0.50, 0.75] ):\n",
    "          QUICK summary statistics on given dataframe.\n",
    "     stats( dataframe ):\n",
    "          VERBOSE statistics on given dataframe; CORRELATIONS without regression.\n",
    "     todf( data, col='Y' ):\n",
    "          CONVERT (list, Series, or DataFrame) TO DataFrame, NAMING single column.\n",
    "     paste( df_list ):\n",
    "          Merge dataframes (not Series) across their common index values.\n",
    "     writefile( dataframe, filename='tmp-yi_1tools.csv', separator=',' ):\n",
    "         Write dataframe to disk file using UTF-8 encoding.\n",
    "\n",
    "#### yi_fred.py\n",
    "\n",
    "     readfile( filename, separator=',', compress=None ):\n",
    "         Read file (CSV default) as pandas dataframe.\n",
    "     makeURL( fredcode ):\n",
    "         Create http address to access FRED's CSV files.\n",
    "     getdata_fred( fredcode ):\n",
    "         Download CSV file from FRED and read it as pandas DATAFRAME.\n",
    "     plotdf( dataframe, title='tmp' ):\n",
    "         Plot dataframe where its index are dates.\n",
    "     daily( dataframe ):\n",
    "          Resample data to daily using only business days.\n",
    "     monthly( dataframe ):\n",
    "          Resample data to FRED's month start frequency.\n",
    "     quarterly( dataframe ):\n",
    "          Resample data to FRED's quarterly start frequency.\n",
    "     getm4eurusd( fredcode=d4eurusd ):\n",
    "          Make monthly EURUSD, and try to prepend 1971-2002 archive.\n",
    "     getspx( fredcode=d4spx ):\n",
    "          Make daily S&P 500 series, and try to prepend 1957-archive.\n",
    "     gethomepx( fredcode=m4homepx ):\n",
    "          Make Case-Shiller 20-city, and try to prepend 1987-2000 10-city.\n",
    "     getinflations( inflations=ml_infl ):\n",
    "          Normalize and average all inflation measures.\n",
    "     getdeflator( inflation=m4infl ):\n",
    "          Construct a de-inflation dataframe suitable as multiplier.\n",
    "     getm4infleu( ):\n",
    "          Normalize and average Eurozone Consumer Prices.\n",
    "     getfred( fredcode ):\n",
    "          Retrieve from FRED in dataframe format, INCL. SPECIAL CASES.\n",
    "     plotfred( data, title='tmp', maxi=87654321 ):\n",
    "          Plot data should be it given as dataframe or fredcode.\n",
    "     holtfred( data, h=24, alpha=ts.hw_alpha, beta=ts.hw_beta ):\n",
    "          Holt-Winters forecast h-periods ahead (fredcode aware).\n",
    "\n",
    "#### yi_plot.py\n",
    "\n",
    "     plotn( dataframe, title='tmp' ):\n",
    "         Plot dataframe where the index is numbered (not dates).\n",
    "     boxplot( data, title='tmp', labels=[] ):\n",
    "          Make boxplot from data which could be a dataframe.\n",
    "     scatter( dataframe, title='tmp', col=[0, 1] ):\n",
    "         Scatter plot for dataframe by zero-based column positions.\n",
    "     scats( dataframe, title='tmp' ):\n",
    "         All pair-wise scatter plots for dataframe.\n",
    "     scat( dfx, dfy, title='tmp', col=[0, 1] ):\n",
    "         Scatter plot between two pasted dataframes.\n",
    "\n",
    "#### yi_quandl.py\n",
    "\n",
    "     setQuandlToken( API_key ):\n",
    "          Generate authtoken.p in the local directory for API access.\n",
    "     cotr_get( futures='GC', type='FO' ):\n",
    "          Get CFTC Commitment of Traders Report COTR.\n",
    "     cotr_position( futures='GC' ):\n",
    "          Extract market position from CFTC Commitment of Traders Report.\n",
    "     cotr_position_usd():\n",
    "          Market position for USD from COTR of JY and EC.\n",
    "     cotr_position_metals():\n",
    "          Market position for precious metals from COTR of GC and SI.\n",
    "     cotr_position_bonds():\n",
    "          Market position for bonds from COTR of TY and ED.\n",
    "     cotr_position_equities():\n",
    "          Market position for equities from COTR of both SP and ES.\n",
    "     fut_decode( slang ):\n",
    "         Validate and translate slang string into vendor futures code.\n",
    "     getfut( slang, maxi=512, col='Settle' ):\n",
    "          slang string retrieves single column for one futures contract.\n",
    "     getqdl( quandlcode, maxi=87654321 ):\n",
    "          Retrieve from Quandl in dataframe format, INCL. SPECIAL CASES.\n",
    "     plotqdl( data, title='tmp', maxi=87654321 ):\n",
    "          Plot data should be it given as dataframe or quandlcode.\n",
    "     holtqdl( data, h=24, alpha=ts.hw_alpha, beta=ts.hw_beta ):\n",
    "          Holt-Winters forecast h-periods ahead (quandlcode aware).\n",
    "\n",
    "#### yi_secform.py\n",
    "\n",
    "     parse13f( url=druck150814 ):\n",
    "          Parse SEC form 13F into a pandas dataframe.\n",
    "     pcent13f( url=druck150814, top=7654321 ):\n",
    "          Prune, then sort SEC 13F by percentage allocation, showing top N.\n",
    "\n",
    "#### yi_simulation.py\n",
    "\n",
    "     GET_simu_spx_pcent():\n",
    "          Retrieve normalized SPX daily percent change 1957-2014.\n",
    "     SHAPE_simu_spx_pcent( mean=MEAN_PC_SPX, std=STD_PC_SPX ):\n",
    "          Generate SPX percent change (defaults are ACTUAL annualized numbers).\n",
    "     SHAPE_simu_spx_returns( mean=MEAN_PC_SPX, std=STD_PC_SPX ):\n",
    "          Convert percent form to return form.\n",
    "     array_spx_returns( mean=MEAN_PC_SPX, std=STD_PC_SPX ):\n",
    "          Array of SPX in return form.\n",
    "     bootstrap( N, yarray ):\n",
    "          Randomly pick out N without replacment from yarray.\n",
    "     simu_prices( N, yarray ):\n",
    "          Convert bootstrap returns to price time-series into pandas DATAFRAME.\n",
    "     simu_plots_spx( charts=1, N=N_PC_SPX, mean=MEAN_PC_SPX, std=STD_PC_SPX ):\n",
    "          Display simulated SPX price charts of N days, given mean and std.\n",
    "\n",
    "#### yi_stocks.py\n",
    "\n",
    "     stock_decode( slang ):\n",
    "         Validate and translate slang string into vendor stock code.\n",
    "     stock_all( slang, maxi=3650 ):\n",
    "          slang string retrieves ALL columns for single stock.\n",
    "     stock_one( slang, maxi=3650, col='Close' ):\n",
    "          slang string retrieves SINGLE column for said stock.\n",
    "     getstock( slang, maxi=3650 ):\n",
    "          Retrieve stock data from Yahoo Finance or Google Finance.\n",
    "\n",
    "#### yi_timeseries.py\n",
    "\n",
    "     holt_winters_growth( y, alpha=hw_alpha, beta=hw_beta ):\n",
    "          Helper for Holt-Winters growth (linear) model using numpy arrays.\n",
    "     holt( data, alpha=hw_alpha, beta=hw_beta ):\n",
    "          Holt-Winters growth (linear) model outputs workout dataframe.\n",
    "     holtlevel( data, alpha=hw_alpha, beta=hw_beta ):\n",
    "          Just smoothed Level dataframe from Holt-Winters growth model.\n",
    "     holtgrow( data, alpha=hw_alpha, beta=hw_beta ):\n",
    "          Just the Growth dataframe from Holt-Winters growth model.\n",
    "     holtpc( data, yearly=256, alpha=hw_alpha, beta=hw_beta ):\n",
    "          Annualized percentage growth dataframe from H-W growth model.\n",
    "     holtforecast( holtdf, h=12 ):\n",
    "          Given a dataframe from holt, forecast ahead h periods.\n",
    "     plotholt( holtdf, h=12 ):\n",
    "          Given a dataframe from holt, plot forecasts h periods ahead.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
