tag:blogger.com,1999:blog-41608824742755371882024-03-14T00:29:12.964-07:00Peer UnreviewedA place where I talk about some of my academic research. Vishal Misrahttp://www.blogger.com/profile/04748385949397825305noreply@blogger.comBlogger13125tag:blogger.com,1999:blog-4160882474275537188.post-78018739986139133072019-11-20T22:32:00.003-08:002019-11-27T08:01:35.841-08:00A short tutorial on the Robust Synthetic Control python library, Part 1: counterfactuals<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
I have posted a couple of blogs on the powerful technique of (multidimensional) Robust Synthetic Control <a href="http://peerunreviewed.blogspot.com/2019/05/mrsc-new-way-to-answer-what-ifs-and-do.html">here</a> and <a href="http://peerunreviewed.blogspot.com/2019/08/who-is-paying-for-tariffs-china-or-us.html">here</a>. In this post I will give a short tutorial on how you can use mRSC to perform your own analysis using the python package my collaborator Jehangir has made <a href="https://github.com/jehangiramjad/tslib">available</a> on github. This posting will be about counterfactual analysis.<br />
<br />
We will work with the canonical example of the synthetic control based counterfactual analysis of the impact <a href="https://en.wikipedia.org/wiki/1988_California_Proposition_99">California's Prop 99</a>. All the data and code is included in the github repository linked above. I will post the python code as run on a Jupyter Notebook, and the "tslib" library referenced above has been downloaded and is available.<br />
<div class="cell border-box-sizing text_cell rendered" style="background-color: white; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; font-family: "helvetica neue", helvetica, arial, sans-serif; margin: 0px; outline: none; padding: 5px; width: 1110px;">
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="text_cell_render border-box-sizing rendered_html" style="border-style: none; box-sizing: border-box; outline: none; overflow: auto hidden; padding: 0.5em 0.5em 0.5em 0.4em; resize: none; width: inherit;">
<div style="box-sizing: border-box;">
<div>
<div class="cell border-box-sizing text_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; caret-color: rgb(0, 0, 0); display: flex; flex-direction: row; font-family: "Helvetica Neue", Helvetica, Arial, sans-serif; font-size: 14px; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="text_cell_render border-box-sizing rendered_html" style="border-style: none; box-sizing: border-box; outline: none; overflow: auto hidden; padding: 0.5em 0.5em 0.5em 0.4em; resize: none; width: inherit;">
<div class="cell border-box-sizing text_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: row; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="text_cell_render border-box-sizing rendered_html" style="border-style: none; box-sizing: border-box; outline: none; overflow: auto hidden; padding: 0.5em 0.5em 0.5em 0.4em; resize: none; width: inherit;">
<div style="box-sizing: border-box;">
Preliminaries: importing the libraries.</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: column; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="input" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
In [1]:</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="input_area" style="background-color: #f7f7f7; background-position: initial initial; background-repeat: initial initial; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython3" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow-wrap: break-word; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all;"><span style="box-sizing: border-box;"></span><span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">sys</span><span class="o" style="box-sizing: border-box; color: #666666;">,</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">os</span>
<span class="n" style="box-sizing: border-box;">sys</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">path</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">append</span><span class="p" style="box-sizing: border-box;">(</span><span class="s2" style="box-sizing: border-box; color: #ba2121;">"../.."</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">sys</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">path</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">append</span><span class="p" style="box-sizing: border-box;">(</span><span class="s2" style="box-sizing: border-box; color: #ba2121;">".."</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">sys</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">path</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">append</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">os</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">getcwd</span><span class="p" style="box-sizing: border-box;">())</span>
<span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">from</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">matplotlib</span> <span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="n" style="box-sizing: border-box;">pyplot</span> <span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">as</span> <span class="n" style="box-sizing: border-box;">plt</span>
<span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">matplotlib.ticker</span> <span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">as</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">ticker</span>
<span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">matplotlib.dates</span> <span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">as</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">mdates</span>
<span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">numpy</span> <span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">as</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">np</span>
<span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">pandas</span> <span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">as</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">pd</span>
<span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">copy</span>
<span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">from</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">tslib.src</span> <span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="n" style="box-sizing: border-box;">tsUtils</span>
<span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">from</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">tslib.src.synthcontrol.syntheticControl</span> <span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="n" style="box-sizing: border-box;">RobustSyntheticControl</span>
<span class="kn" style="box-sizing: border-box; color: green; font-weight: bold;">from</span> <span class="nn" style="box-sizing: border-box; color: blue; font-weight: bold;">tslib.tests</span> <span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">import</span> <span class="n" style="box-sizing: border-box;">testdata</span>
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: row; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="text_cell_render border-box-sizing rendered_html" style="border-style: none; box-sizing: border-box; outline: none; overflow: auto hidden; padding: 0.5em 0.5em 0.5em 0.4em; resize: none; width: inherit;">
<div style="box-sizing: border-box;">
Reading in the data and converting it into a usable form</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: column; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="input" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
In [2]:</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="input_area" style="background-color: #f7f7f7; background-position: initial initial; background-repeat: initial initial; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython3" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow-wrap: break-word; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all;"><span style="box-sizing: border-box;"></span><span class="n" style="box-sizing: border-box;">directory</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">os</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">path</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">dirname</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">testdata</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="vm" style="box-sizing: border-box; color: #19177c;">__file__</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">filename</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">directory</span> <span class="o" style="box-sizing: border-box; color: #666666;">+</span> <span class="s1" style="box-sizing: border-box; color: #ba2121;">'/prop99.csv'</span>
<span class="n" style="box-sizing: border-box;">df</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">pd</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">read_csv</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">filename</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">df</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">df</span><span class="p" style="box-sizing: border-box;">[</span><span class="n" style="box-sizing: border-box;">df</span><span class="p" style="box-sizing: border-box;">[</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'SubMeasureDesc'</span><span class="p" style="box-sizing: border-box;">]</span> <span class="o" style="box-sizing: border-box; color: #666666;">==</span> <span class="s1" style="box-sizing: border-box; color: #ba2121;">'Cigarette Consumption (Pack Sales Per Capita)'</span><span class="p" style="box-sizing: border-box;">]</span>
<span class="n" style="box-sizing: border-box;">pivot</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">df</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">pivot_table</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">values</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'Data_Value'</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">index</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'LocationDesc'</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">columns</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="p" style="box-sizing: border-box;">[</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'Year'</span><span class="p" style="box-sizing: border-box;">])</span>
<span class="n" style="box-sizing: border-box;">dfProp99</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">pd</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">DataFrame</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">pivot</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">to_records</span><span class="p" style="box-sizing: border-box;">())</span>
<span class="n" style="box-sizing: border-box;">allColumns</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">dfProp99</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">columns</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">values</span>
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: row; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="text_cell_render border-box-sizing rendered_html" style="border-style: none; box-sizing: border-box; outline: none; overflow: auto hidden; padding: 0.5em 0.5em 0.5em 0.4em; resize: none; width: inherit;">
<div style="box-sizing: border-box;">
The code above extracts out the relevant portion for our analysis, and finally what we need looks like the following. This is the format that you should get your dataset into for analysis. One row per "unit", and one column for every timepoint.</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: column; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="input" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
In [3]:</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="input_area" style="background-color: #f7f7f7; background-position: initial initial; background-repeat: initial initial; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython3" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow-wrap: break-word; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all;"><span style="box-sizing: border-box;"></span><span class="n" style="box-sizing: border-box;">pd</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">set_option</span><span class="p" style="box-sizing: border-box;">(</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'display.max_columns'</span><span class="p" style="box-sizing: border-box;">,</span> <span class="mi" style="box-sizing: border-box; color: #666666;">10</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">dfProp99</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">head</span><span class="p" style="box-sizing: border-box;">()</span>
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; position: relative; z-index: 1;">
<div class="output" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column;">
<div class="output_area" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row; padding: 0px;">
<div class="prompt output_prompt" style="-webkit-user-select: none; box-sizing: border-box; color: #d84315; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
Out[3]:</div>
<div class="output_html rendered_html output_subarea output_execute_result" style="-webkit-box-flex: 1; box-sizing: border-box; flex: 1 1 0%; max-width: calc(100% - 14ex); overflow-x: auto; padding: 0.4em;">
<div style="box-sizing: border-box;">
<table border="1" class="dataframe" style="background-color: transparent; border-collapse: collapse; border-spacing: 0px; border: 1px solid black; box-sizing: border-box; margin-left: 0px; margin-right: 0px; margin-top: 1em;"><thead style="box-sizing: border-box;">
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; text-align: right;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;"></th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">LocationDesc</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">1970</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">1971</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">1972</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">1973</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">...</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">2010</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">2011</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">2012</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">2013</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">2014</th></tr>
</thead><tbody style="box-sizing: border-box;">
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">0</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">Alabama</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">89.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">95.4</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">101.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">102.9</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">71.5</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">68.4</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">67.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">64.6</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">61.7</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">1</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">Alaska</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">121.3</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">123.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">130.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">125.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">43.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">43.3</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">41.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">39.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">37.2</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">2</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">Arizona</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">115.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">109.6</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">125.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">128.3</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">24.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">27.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">25.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">24.4</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">23.0</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">3</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">Arkansas</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">100.3</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">104.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">103.9</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">108.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">63.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">61.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">60.5</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">57.5</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">54.4</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">4</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">California</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">123.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">121.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">123.5</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">124.4</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">26.3</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">26.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">25.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">23.9</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">22.7</td></tr>
</tbody></table>
<div style="box-sizing: border-box; margin-top: 1em;">
5 rows × 46 columns</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: row; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="text_cell_render border-box-sizing rendered_html" style="border-style: none; box-sizing: border-box; outline: none; overflow: auto hidden; padding: 0.5em 0.5em 0.5em 0.4em; resize: none; width: inherit;">
<div style="box-sizing: border-box;">
Now we extract the list of donor pools ("otherStates"), the intervention unit ("caStateKey"), the start and end year for the dataset and the year of the intervention.</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: column; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="input" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
In [4]:</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="input_area" style="background-color: #f7f7f7; background-position: initial initial; background-repeat: initial initial; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython3" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow-wrap: break-word; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all;"><span style="box-sizing: border-box;"></span><span class="n" style="box-sizing: border-box;">states</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="nb" style="box-sizing: border-box; color: green;">list</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">np</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">unique</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">dfProp99</span><span class="p" style="box-sizing: border-box;">[</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'LocationDesc'</span><span class="p" style="box-sizing: border-box;">]))</span>
<span class="n" style="box-sizing: border-box;">years</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">np</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">delete</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">allColumns</span><span class="p" style="box-sizing: border-box;">,</span> <span class="p" style="box-sizing: border-box;">[</span><span class="mi" style="box-sizing: border-box; color: #666666;">0</span><span class="p" style="box-sizing: border-box;">])</span>
<span class="n" style="box-sizing: border-box;">caStateKey</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="s1" style="box-sizing: border-box; color: #ba2121;">'California'</span>
<span class="n" style="box-sizing: border-box;">states</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">remove</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">caStateKey</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">otherStates</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">states</span>
<span class="n" style="box-sizing: border-box;">yearStart</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="mi" style="box-sizing: border-box; color: #666666;">1970</span>
<span class="n" style="box-sizing: border-box;">yearTrainEnd</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="mi" style="box-sizing: border-box; color: #666666;">1989</span>
<span class="n" style="box-sizing: border-box;">yearTestEnd</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="mi" style="box-sizing: border-box; color: #666666;">2015</span>
<span class="n" style="box-sizing: border-box;">p</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="mf" style="box-sizing: border-box; color: #666666;">1.0</span>
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: row; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="text_cell_render border-box-sizing rendered_html" style="border-style: none; box-sizing: border-box; outline: none; overflow: auto hidden; padding: 0.5em 0.5em 0.5em 0.4em; resize: none; width: inherit;">
<div style="box-sizing: border-box;">
Now we prep the data and split it into training and test sets.</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: column; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="input" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
In [5]:</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="input_area" style="background-color: #f7f7f7; background-position: initial initial; background-repeat: initial initial; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython3" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow-wrap: break-word; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all;"><span style="box-sizing: border-box;"></span><span class="n" style="box-sizing: border-box;">trainingYears</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="p" style="box-sizing: border-box;">[]</span>
<span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">for</span> <span class="n" style="box-sizing: border-box;">i</span> <span class="ow" style="box-sizing: border-box; color: #aa22ff; font-weight: bold;">in</span> <span class="nb" style="box-sizing: border-box; color: green;">range</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">yearStart</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">yearTrainEnd</span><span class="p" style="box-sizing: border-box;">,</span> <span class="mi" style="box-sizing: border-box; color: #666666;">1</span><span class="p" style="box-sizing: border-box;">):</span>
<span class="n" style="box-sizing: border-box;">trainingYears</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">append</span><span class="p" style="box-sizing: border-box;">(</span><span class="nb" style="box-sizing: border-box; color: green;">str</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">i</span><span class="p" style="box-sizing: border-box;">))</span>
<span class="n" style="box-sizing: border-box;">testYears</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="p" style="box-sizing: border-box;">[]</span>
<span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">for</span> <span class="n" style="box-sizing: border-box;">i</span> <span class="ow" style="box-sizing: border-box; color: #aa22ff; font-weight: bold;">in</span> <span class="nb" style="box-sizing: border-box; color: green;">range</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">yearTrainEnd</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">yearTestEnd</span><span class="p" style="box-sizing: border-box;">,</span> <span class="mi" style="box-sizing: border-box; color: #666666;">1</span><span class="p" style="box-sizing: border-box;">):</span>
<span class="n" style="box-sizing: border-box;">testYears</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">append</span><span class="p" style="box-sizing: border-box;">(</span><span class="nb" style="box-sizing: border-box; color: green;">str</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">i</span><span class="p" style="box-sizing: border-box;">))</span>
<span class="n" style="box-sizing: border-box;">trainDataMasterDict</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="p" style="box-sizing: border-box;">{}</span>
<span class="n" style="box-sizing: border-box;">trainDataDict</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="p" style="box-sizing: border-box;">{}</span>
<span class="n" style="box-sizing: border-box;">testDataDict</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="p" style="box-sizing: border-box;">{}</span>
<span class="k" style="box-sizing: border-box; color: green; font-weight: bold;">for</span> <span class="n" style="box-sizing: border-box;">key</span> <span class="ow" style="box-sizing: border-box; color: #aa22ff; font-weight: bold;">in</span> <span class="n" style="box-sizing: border-box;">otherStates</span><span class="p" style="box-sizing: border-box;">:</span>
<span class="n" style="box-sizing: border-box;">series</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">dfProp99</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">loc</span><span class="p" style="box-sizing: border-box;">[</span><span class="n" style="box-sizing: border-box;">dfProp99</span><span class="p" style="box-sizing: border-box;">[</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'LocationDesc'</span><span class="p" style="box-sizing: border-box;">]</span> <span class="o" style="box-sizing: border-box; color: #666666;">==</span> <span class="n" style="box-sizing: border-box;">key</span><span class="p" style="box-sizing: border-box;">]</span>
<span class="n" style="box-sizing: border-box;">trainDataMasterDict</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">update</span><span class="p" style="box-sizing: border-box;">({</span><span class="n" style="box-sizing: border-box;">key</span><span class="p" style="box-sizing: border-box;">:</span> <span class="n" style="box-sizing: border-box;">series</span><span class="p" style="box-sizing: border-box;">[</span><span class="n" style="box-sizing: border-box;">trainingYears</span><span class="p" style="box-sizing: border-box;">]</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">values</span><span class="p" style="box-sizing: border-box;">[</span><span class="mi" style="box-sizing: border-box; color: #666666;">0</span><span class="p" style="box-sizing: border-box;">]})</span>
<span class="c1" style="box-sizing: border-box; color: #408080; font-style: italic;"># randomly hide training data</span>
<span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">trainData</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">pObservation</span><span class="p" style="box-sizing: border-box;">)</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">tsUtils</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">randomlyHideValues</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">copy</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">deepcopy</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">trainDataMasterDict</span><span class="p" style="box-sizing: border-box;">[</span><span class="n" style="box-sizing: border-box;">key</span><span class="p" style="box-sizing: border-box;">]),</span> <span class="n" style="box-sizing: border-box;">p</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">trainDataDict</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">update</span><span class="p" style="box-sizing: border-box;">({</span><span class="n" style="box-sizing: border-box;">key</span><span class="p" style="box-sizing: border-box;">:</span> <span class="n" style="box-sizing: border-box;">trainData</span><span class="p" style="box-sizing: border-box;">})</span>
<span class="n" style="box-sizing: border-box;">testDataDict</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">update</span><span class="p" style="box-sizing: border-box;">({</span><span class="n" style="box-sizing: border-box;">key</span><span class="p" style="box-sizing: border-box;">:</span> <span class="n" style="box-sizing: border-box;">series</span><span class="p" style="box-sizing: border-box;">[</span><span class="n" style="box-sizing: border-box;">testYears</span><span class="p" style="box-sizing: border-box;">]</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">values</span><span class="p" style="box-sizing: border-box;">[</span><span class="mi" style="box-sizing: border-box; color: #666666;">0</span><span class="p" style="box-sizing: border-box;">]})</span>
<span class="n" style="box-sizing: border-box;">series</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">dfProp99</span><span class="p" style="box-sizing: border-box;">[</span><span class="n" style="box-sizing: border-box;">dfProp99</span><span class="p" style="box-sizing: border-box;">[</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'LocationDesc'</span><span class="p" style="box-sizing: border-box;">]</span> <span class="o" style="box-sizing: border-box; color: #666666;">==</span> <span class="n" style="box-sizing: border-box;">caStateKey</span><span class="p" style="box-sizing: border-box;">]</span>
<span class="n" style="box-sizing: border-box;">trainDataMasterDict</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">update</span><span class="p" style="box-sizing: border-box;">({</span><span class="n" style="box-sizing: border-box;">caStateKey</span><span class="p" style="box-sizing: border-box;">:</span> <span class="n" style="box-sizing: border-box;">series</span><span class="p" style="box-sizing: border-box;">[</span><span class="n" style="box-sizing: border-box;">trainingYears</span><span class="p" style="box-sizing: border-box;">]</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">values</span><span class="p" style="box-sizing: border-box;">[</span><span class="mi" style="box-sizing: border-box; color: #666666;">0</span><span class="p" style="box-sizing: border-box;">]})</span>
<span class="n" style="box-sizing: border-box;">trainDataDict</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">update</span><span class="p" style="box-sizing: border-box;">({</span><span class="n" style="box-sizing: border-box;">caStateKey</span><span class="p" style="box-sizing: border-box;">:</span> <span class="n" style="box-sizing: border-box;">series</span><span class="p" style="box-sizing: border-box;">[</span><span class="n" style="box-sizing: border-box;">trainingYears</span><span class="p" style="box-sizing: border-box;">]</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">values</span><span class="p" style="box-sizing: border-box;">[</span><span class="mi" style="box-sizing: border-box; color: #666666;">0</span><span class="p" style="box-sizing: border-box;">]})</span>
<span class="n" style="box-sizing: border-box;">testDataDict</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">update</span><span class="p" style="box-sizing: border-box;">({</span><span class="n" style="box-sizing: border-box;">caStateKey</span><span class="p" style="box-sizing: border-box;">:</span> <span class="n" style="box-sizing: border-box;">series</span><span class="p" style="box-sizing: border-box;">[</span><span class="n" style="box-sizing: border-box;">testYears</span><span class="p" style="box-sizing: border-box;">]</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">values</span><span class="p" style="box-sizing: border-box;">[</span><span class="mi" style="box-sizing: border-box; color: #666666;">0</span><span class="p" style="box-sizing: border-box;">]})</span>
<span class="n" style="box-sizing: border-box;">trainMasterDF</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">pd</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">DataFrame</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">data</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="n" style="box-sizing: border-box;">trainDataMasterDict</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">trainDF</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">pd</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">DataFrame</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">data</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="n" style="box-sizing: border-box;">trainDataDict</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">testDF</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">pd</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">DataFrame</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">data</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="n" style="box-sizing: border-box;">testDataDict</span><span class="p" style="box-sizing: border-box;">)</span>
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: row; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="text_cell_render border-box-sizing rendered_html" style="border-style: none; box-sizing: border-box; outline: none; overflow: auto hidden; padding: 0.5em 0.5em 0.5em 0.4em; resize: none; width: inherit;">
<div style="box-sizing: border-box;">
This is what the test and train matrices look like (in the library we represent each donor pool or intervention unit as a separate column, and each timepoint as a row)</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: column; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="input" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
In [6]:</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="input_area" style="background-color: #f7f7f7; background-position: initial initial; background-repeat: initial initial; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython3" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow-wrap: break-word; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all;"><span style="box-sizing: border-box;"></span><span class="n" style="box-sizing: border-box;">trainDF</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">head</span><span class="p" style="box-sizing: border-box;">()</span>
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; position: relative; z-index: 1;">
<div class="output" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column;">
<div class="output_area" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row; padding: 0px;">
<div class="prompt output_prompt" style="-webkit-user-select: none; box-sizing: border-box; color: #d84315; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
Out[6]:</div>
<div class="output_html rendered_html output_subarea output_execute_result" style="-webkit-box-flex: 1; box-sizing: border-box; flex: 1 1 0%; max-width: calc(100% - 14ex); overflow-x: auto; padding: 0.4em;">
<div style="box-sizing: border-box;">
<table border="1" class="dataframe" style="background-color: transparent; border-collapse: collapse; border-spacing: 0px; border: 1px solid black; box-sizing: border-box; margin-left: 0px; margin-right: 0px; margin-top: 1em;"><thead style="box-sizing: border-box;">
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; text-align: right;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;"></th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Alabama</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Alaska</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Arizona</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Arkansas</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Colorado</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">...</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Washington</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">West Virginia</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Wisconsin</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Wyoming</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">California</th></tr>
</thead><tbody style="box-sizing: border-box;">
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">0</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">89.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">121.3</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">115.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">100.3</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">124.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">96.7</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">114.5</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">106.4</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">132.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">123.0</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">1</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">95.4</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">123.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">109.6</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">104.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">125.5</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">97.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">111.5</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">105.4</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">131.7</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">121.0</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">2</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">101.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">130.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">125.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">103.9</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">134.3</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">88.5</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">117.5</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">108.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">140.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">123.5</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">3</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">102.9</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">125.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">128.3</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">108.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">137.9</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">91.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">116.6</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">109.5</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">141.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">124.4</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">4</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">108.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">130.4</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">133.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">109.7</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">132.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">98.6</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">119.9</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">111.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">145.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">126.7</td></tr>
</tbody></table>
<div style="box-sizing: border-box; margin-top: 1em;">
5 rows × 51 columns</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: column; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="input" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
In [7]:</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="input_area" style="background-color: #f7f7f7; background-position: initial initial; background-repeat: initial initial; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython3" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow-wrap: break-word; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all;"><span style="box-sizing: border-box;"></span><span class="n" style="box-sizing: border-box;">testDF</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">head</span><span class="p" style="box-sizing: border-box;">()</span>
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; position: relative; z-index: 1;">
<div class="output" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column;">
<div class="output_area" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row; padding: 0px;">
<div class="prompt output_prompt" style="-webkit-user-select: none; box-sizing: border-box; color: #d84315; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
Out[7]:</div>
<div class="output_html rendered_html output_subarea output_execute_result" style="-webkit-box-flex: 1; box-sizing: border-box; flex: 1 1 0%; max-width: calc(100% - 14ex); overflow-x: auto; padding: 0.4em;">
<div style="box-sizing: border-box;">
<table border="1" class="dataframe" style="background-color: transparent; border-collapse: collapse; border-spacing: 0px; border: 1px solid black; box-sizing: border-box; margin-left: 0px; margin-right: 0px; margin-top: 1em;"><thead style="box-sizing: border-box;">
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; text-align: right;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;"></th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Alabama</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Alaska</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Arizona</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Arkansas</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Colorado</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">...</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Washington</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">West Virginia</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Wisconsin</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">Wyoming</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: right; vertical-align: middle;">California</th></tr>
</thead><tbody style="box-sizing: border-box;">
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">0</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">105.6</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">94.4</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">96.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">118.3</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">88.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">86.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">104.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">100.3</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">111.4</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">82.4</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">1</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">108.6</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">100.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">88.9</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">113.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">87.4</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">83.4</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">104.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">94.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">96.9</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">77.8</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">2</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">107.9</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">101.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">81.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">116.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">90.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">78.7</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">100.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">95.5</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">109.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">68.7</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">3</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">109.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">98.5</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">79.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">126.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">88.3</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">81.1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">98.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">96.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">110.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">67.5</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">4</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">108.5</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">95.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">80.3</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">113.8</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">88.6</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">...</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">79.4</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">111.0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">91.2</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">108.4</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">63.4</td></tr>
</tbody></table>
<div style="box-sizing: border-box; margin-top: 1em;">
5 rows × 51 columns</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: row; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="text_cell_render border-box-sizing rendered_html" style="border-style: none; box-sizing: border-box; outline: none; overflow: auto hidden; padding: 0.5em 0.5em 0.5em 0.4em; resize: none; width: inherit;">
<div style="box-sizing: border-box;">
Next, we do a sanity check to see if the matrix is indeed low rank. It checks out and looks like we will need only about 4 singular values.</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: column; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="input" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
In [8]:</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="input_area" style="background-color: #f7f7f7; background-position: initial initial; background-repeat: initial initial; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython3" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow-wrap: break-word; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all;"><span style="box-sizing: border-box;"></span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">U</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">s</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">Vh</span><span class="p" style="box-sizing: border-box;">)</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">np</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">linalg</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">svd</span><span class="p" style="box-sizing: border-box;">((</span><span class="n" style="box-sizing: border-box;">trainDF</span><span class="p" style="box-sizing: border-box;">)</span> <span class="o" style="box-sizing: border-box; color: #666666;">-</span> <span class="n" style="box-sizing: border-box;">np</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">mean</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">trainDF</span><span class="p" style="box-sizing: border-box;">))</span>
<span class="n" style="box-sizing: border-box;">s2</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">np</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">power</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">s</span><span class="p" style="box-sizing: border-box;">,</span> <span class="mi" style="box-sizing: border-box; color: #666666;">2</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">spectrum</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">np</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">cumsum</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">s2</span><span class="p" style="box-sizing: border-box;">)</span><span class="o" style="box-sizing: border-box; color: #666666;">/</span><span class="n" style="box-sizing: border-box;">np</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">sum</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">s2</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">plot</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">spectrum</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">grid</span><span class="p" style="box-sizing: border-box;">()</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">title</span><span class="p" style="box-sizing: border-box;">(</span><span class="s2" style="box-sizing: border-box; color: #ba2121;">"Cumulative energy"</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">figure</span><span class="p" style="box-sizing: border-box;">()</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">plot</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">s2</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">grid</span><span class="p" style="box-sizing: border-box;">()</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">xlabel</span><span class="p" style="box-sizing: border-box;">(</span><span class="s2" style="box-sizing: border-box; color: #ba2121;">"Ordered Singular Values"</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">ylabel</span><span class="p" style="box-sizing: border-box;">(</span><span class="s2" style="box-sizing: border-box; color: #ba2121;">"Energy"</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">title</span><span class="p" style="box-sizing: border-box;">(</span><span class="s2" style="box-sizing: border-box; color: #ba2121;">"Singular Value Spectrum"</span><span class="p" style="box-sizing: border-box;">)</span>
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; position: relative; z-index: 1;">
<div class="output" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column;">
<div class="output_area" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row; padding: 0px;">
<div class="prompt output_prompt" style="-webkit-user-select: none; box-sizing: border-box; color: #d84315; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
Out[8]:</div>
<div class="output_text output_subarea output_execute_result" style="-webkit-box-flex: 1; box-sizing: border-box; flex: 1 1 0%; line-height: 1.21429em; max-width: calc(100% - 14ex); overflow-x: auto; padding: 0.4em;">
<pre style="background-color: transparent; border-radius: 0px; border: 0px; box-sizing: border-box; font-size: inherit; line-height: inherit; overflow-wrap: break-word; overflow: auto; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-break: break-all;">Text(0.5,1,'Singular Value Spectrum')</pre>
</div>
</div>
<div class="output_area" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row; padding: 0px;">
<div class="prompt" style="-webkit-user-select: none; box-sizing: border-box; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0px 0.4em; text-align: right;">
</div>
<div class="output_png output_subarea " style="-webkit-box-flex: 1; box-sizing: border-box; flex: 1 1 0%; max-width: calc(100% - 14ex); overflow-x: auto; padding: 0.4em;">
<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XucXVV99/HPd+5JJvfAkJBwNfgYBIGMQcTaSRWItBXr7RWqiFabtop9qk+tYPsgL6yttrV91FIt1XjXYLFi2sZirBmlCpggASEYCOGSYSYBksyESWYyt9/zx96TnEzOzJzMnLmd/X2/Xud19mWtc35n5+R39qy911qKCMzMLBvKJjoAMzMbP076ZmYZ4qRvZpYhTvpmZhnipG9mliFO+mZmGeKkbyVJ0k2Svj6K+g9LaihiSGaTgpO+FZWk35W0RVK7pBZJ35f0qomOayiSvizpL3O3RcS5EdE4QSGZjRknfSsaSR8E/h/wV0AdcBrwT8BVExlX1ijh/9uWl78YVhSSZgM3A++LiH+LiIMR0R0R/x4RH0rLHHNGLalBUlPO+pOSPiTpQUkHJX1RUl3618ILkn4oaW6+ujn1XztIfP8qabekNkk/kXRuun0N8Dbgz9K/Tv4997UkLZLUIWlezmtdKOl5SZXp+u9JekTSfkl3Sjp9iOP0Ckk/k9Qq6YHcJiRJjZI+Jumn6ef9gaQFJ1D345J+ChwCzpJ0ZvpZ+4/dLf1NXpL+U9L7B8T2oKQ3DBa7lQYnfSuWS4Aa4LujfJ03AZcB5wC/DXwf+AiwgOT7+scjfN3vA0uBk4FfAN8AiIhb0+W/iYjaiPjt3EoR0QzcncbV73eB2yOiO02SHwHeCJwE3AV8K18Akk4F/hP4S2Ae8KfAdySdNOC135XGWZWWKbTuNcAaYCbwFPBN4OfAfOCmdH+/rwBvz4ntZcCpwIZ8sVvpcNK3YpkPPB8RPaN8nc9GxJ6IeIYkgd4bEfdHxGGSH5QLR/KiEbE2Il5IX+cm4GXpXyeF+CZwNSRNJ8DqdBvAHwB/HRGPpJ/9r4ALBjnbfzuwISI2RERfRGwEtgBX5pT5UkQ8GhEdwLeBC06g7pcj4uE0joXAy4EbI6IrIv4HWJ9T9nvAUklL0/VrgNsioqvAY2JTlJO+FcteYIGkilG+zp6c5Y4867Un+oKSyiV9QtLjkg4AT6a7FgxRLdftwCWSFgGvBoLkBwngdODTaZNLK7APEMlZ80CnA2/pL5uWfxVJgu63O2f5EEc/byF1d+UsLwL2RcShfPvTH79vA29P2/+vBr423IGwqW+0/0HN+t0NdAJvIEmS+RwEpuesnzKK9zvmtSSVkzSv5PO7JBeTX0uS8GcD+0mSMyRJfFAR0SrpB8BbgZcA34qjw9PuAj4eEd8oIOZdwNci4vcLKDuSurmfowWYJ2l6TuJfMqD8V0gS/f8AhyLi7hHEZVOMz/StKCKiDbgRuEXSGyRNl1Qp6XWS/iYtthW4UtI8SacAfzKKt3wUqJH0m+kF1b8AqgcpOxM4TPLXyHSSJphce4Czhnm/bwLvIGnb/2bO9s8DN+RcGJ4t6S2DvMbXgd+WdEX610dNekF68TDvfcJ1I+IpkuafmyRVSbqE5BpJbpm7gT7gU/gsPzOc9K1oIuLvgQ+SJODnSM5OrwPuSIt8DXiA5Gz7B8Bto3ivNuC9wBeAZ0jO/JsGKf5VkgubzwDbgHsG7P8isCxtNrljYOXUepILwXsi4oGcOL4LfBJYlzYdPQS8bpCYd5H8xfERjh6fD1HA/8MR1n0byQX2vSQXgG8j+fHL9VXgPJIfFcsAeRIVs2yQdBvwq4j4aM62dwBrImJSd6Cz4vGZvlmJkvRySWdLKpO0iuQvhTty9k8n+Wvp1omK0cafk75Z6ToFaATagc8AfxQR9wNIuoKkmWgPx16jsBLn5h0zswzxmb6ZWYZMuvv0FyxYEGecccaI6x88eJAZM2YUL6Ax4jiLa6rECVMnVsdZfGMZ63333fd8RAzWV+WoiJhUj+XLl8dobNq0aVT1x4vjLK6pEmfE1InVcRbfWMYKbIkCcqybd8zMMsRJ38wsQ5z0zcwyxEnfzCxDnPTNzDJk2KQvaa2kZyU9NMh+SfqMpB3pdGsX5ey7VtJj6ePaYgZuZmYnrpAz/S8Dq4bY/zqS0QeXkkzV9jmAdE7RjwIXAyuAjyqd39TMzCbGsJ2zIuInks4YoshVwFfT+0TvkTRH0kKgAdgYEfsAJG0k+fHIO3+omdlAEUFfQHdvH719QU9f0JMud/cFvb3B7oN9PLbnBXoj6O07+uiLoLcPevr66OuD3gj60tc4uv/Y5QjoS9+zN72vva8vWU+2H12O4EjdvgAKGNLmyae6+EXX9kH3nzJ7Gr978WlFPILHK0aP3FM5dpq2pnTbYNuPI2kNyV8J1NXV0djYOOJg2tvbR1V/vDjO4poqccLEx9oXQXcfdPdCT1+y3NMHPQHdfZEs98ELBzu477YfpvuT7f1l+8v19kFPmlx7I3kk2+PY9ZwyPf1l0zI9fRxJsr2RLue8XkHu+smYHrNCaPgiQMDjOwbde9bsMhZ17CxWSHkVI+nn+6wxxPbjN0bcSjq8a319fTQ0NIw4mMbGRkZTf7w4zuKaKnHC4LH29gWHunro6O6lo6uXQ+mjs7t/uYeOrt5kf3cvnd19HO5O9nd296Xbeuns6aOzuzfdl7M93dfV01dgpOL4OVeOVVkuKsvLqChLn9P1/m0V5WVUVYmqdL2q4uj2ynJRUZbWKSujvFzJvnRbspyULc9ZTp6Plt2+/Vecd+4yystEmZLt5WWirEyUK1lOHlBeVka5RFkZR16zTMmjvExIHHkdiWR7ul9lR9f795Wl5aXCUv5k+J4WI+k3cezcm4uB5nR7w4DtjUV4P7MJFRF0dvdxsKuHg4d7OHg4Scjth3s41NWbbuvhYJqoDx5Ot6XLLc918He/vCtJ6F29HEqTeuHJ+KjqijJqKsupqUyfK8qpqSqnpqKMOdOrjtk+raqc6sqypExlOVUVZVRVlFFdnj6n61UVZVSVl/HQg1t5xYr6ZHv50fL9+yvLC092Y6nxhR00vGzRRIcxZRQj6a8HrpO0juSibVtEtEi6E/irnIu3lwM3FOH9zEasP2Ef6Ozmhc5u2jp60uUeDnSkz+m+Ax09yXNnz5H1/uTdV2CzQ3mZmFFVzozqCqanz2WCk2qrmV5VwbSqcqZXJQl5WmX/cgXTK9NtVeVMryxPy5Yd2VdTWU51RRllZWOXdDueLufcRbPH7PVtYgyb9CV9i+SMfYGkJpI7cioBIuLzwAbgSmAHcAh4V7pvn6SPAZvTl7q5/6KuWbF09fSx/1AXTx3o5a7HnmPfwS72tnex9+DhI8v7Dnax92AXbR3dHOjopmeYjF1RJmZNq2RWTQWzplUys6aCk2fWMrOmgtrqSmZUJ0m4Nn2eUd2f1NPlqoojSb66ouy4s+HkT/wVY3lYzAZVyN07Vw+zP4D3DbJvLbB2ZKFZVvX1BXsPdtHS1kFLWyctrR081344TeZJEt93sIvn2w/zQmfP0Yo/+/mRxTLBvBnVzJ9RxbwZVSxbNIu50yuZVVPJzJpKZk2rSJ7TxD6rpuLIvprK4xO1WamYdOPpW2nrT+i72zppbus45rmltZOWAx3saTtMV++x7dvlZWLu9Crmz6hifm0V5y6alSb0aubXVtHy5GO8esWFzK9NEv3saZVj2vRhNlU56VtRHe7ppbm1k6b9h9i1r4Om/Yd4prVjyIReWS5OmV3DwlnTuOi0uZwyu4ZFs6cd8zx/RtWQSbyx8wkuPmv+WH88synPSd9OSE9vHy1tnezaf4im/R007Uued6VJfs8Lncf0Uako05HkfeGSuSw8r4aFs2pYOGcaC2fXsHD2tGETupkVj5O+5fXsgU7u39XKf+3o4j+fe+BIYm9p66Q350KoBAtn1bB43nRe+aL5LJk7nSXzprN47jSWzJtO3cxqKso9rp/ZZOGkb3R29/LLZ9rY+nQrW3clj2daO47sP3nmcyyZN53lp89lydyjCX3x3GksnD2NqgondbOpwkk/Y/r6gif2HuT+p1vZums/W3e18quWF47cxnjqnGlccNoc3nXpGVx42hye3/EAV7xm5QRHbWbF4qRf4vYd7EqS+9Ot3L+rlQd2tXIgvc2xtrqC8xfP5g9+/SwuWDKXC5bM4aSZ1cfUb3zCbe1mpcRJv8Qc7unl7sf3snHbHv5nx/M8tfcQkNy3fk7dTH7z/EVcuGQOF5w2h7NPqqXcF1DNMsVJvwS0HuriR796lh8+socfb3+Og129TK8q59IXLeDqFadxwZI5nHfqbGZU+5/bLOucBaaop/YeZOO2PWzctoctT+2nty84eWY1r7/gVC5fVsclZ8+nprJ8osM0s0nGSX+K6OsLHmhq5YePJIn+0T3tALy4biZ/9Otn89pldZx/6mzf725mQ3LSn8Q6u3v52ePPs3HbHn74yLM898JhysvEijPm8X9/6zQue0kdp82fPtFhmtkU4qQ/yfT1BT9+9DnWbX6anzz6PB3dvcyoKqfhxSdz2bI6Gl58EnOmV010mGY2RTnpTxKHunr4t188w9qfPsHO5w5y0sxq3rT8VC5bdgqvOGse1RVunzez0XPSn2C72zr5yt1P8s17n6ato5vzF8/m06sv4MrzFlLp4QvMrMic9CfIE229fHfd/fzngy30RXD5slN496+dSf3pcz2Wu5mNGSf9cdTbF2zctpsv/s8TbH6yk9rqZ3nHJWfwrkvPYMk8X5A1s7FXUNKXtAr4NFAOfCEiPjFg/+kkM2SdBOwD3h4RTem+XuCXadGnI+L1RYp9ynihs5tvb2niyz97gl37Olg8dxpX/68qPrK6gZk1lRMdnpllSCFz5JYDtwCXAU3AZknrI2JbTrG/A74aEV+R9BvAXwPXpPs6IuKCIsc9Jezad4gv/fRJvr1lF+2He3j5GXP58ytfwmXLTuGun/zYCd/Mxl0hZ/orgB0RsRNA0jrgKiA36S8DPpAubwLuKGaQU82WJ/fxhbue4AfbdlMm8ZvnL+TdrzqT8xfPmejQzCzjFLnTHOUrIL0ZWBUR70nXrwEujojrcsp8E7g3Ij4t6Y3Ad4AFEbFXUg+wFegBPhERx/0gSFoDrAGoq6tbvm7duhF/oPb2dmpra0dcf7Q2PtXNNx7pYkYlNCyu5DWnVzCv5vi7cCY6zkI5zuKbKrE6zuIby1hXrlx5X0TUD1swIoZ8AG8hacfvX78G+OyAMouAfwPuJ2n7bwJm9+9Ln88CngTOHur9li9fHqOxadOmUdUfjTsfaokzrv+PeM9XNsfBw91Dlp3IOE+E4yy+qRKr4yy+sYwV2BLD5POIKKh5pwlYkrO+GGge8MPRDLwRQFIt8KaIaMvZR0TslNQIXAg8XsD7Tilbd7Xyx+vu5/zFc/jM6guZVuXOVGY2+RTS+2czsFTSmZKqgNXA+twCkhZI6n+tG0ju5EHSXEnV/WWASzn2WkBJ2LXvEO/5ymZOmlnNF6+td8I3s0lr2KQfET3AdcCdwCPAtyPiYUk3S+q//bIB2C7pUaAO+Hi6/SXAFkkPkFzg/UQce9fPlNd6qItrv/RzunuDL79rBQtqq4evZGY2QQq6Tz8iNgAbBmy7MWf5duD2PPV+Bpw3yhgnrcM9vfzB1+6jaV8HX3v3Cs4+aWpcTDKz7HKP3BHq6wv+7PYHufeJfXx69QVcfNb8iQ7JzGxYHtFrhD61cTvf29rMh654MVddcOpEh2NmVhAn/RFY9/OnuWXT41y9YgnvbTh7osMxMyuYk/4J+vGjz/HndzzEr59zEh+76qUeEdPMphQn/ROwrfkA7/36fZxTN5Nb3nYRFR7v3symGGetArW0dfB7X97MrGmVfOmdL6e22tfAzWzqceYqwAud3bzrS5tpP9zDv/7hJZwyu2aiQzIzGxGf6Q+ju7eP937jF+x4tp3Pvf0iXrJw1kSHZGY2Yj7TH0JE8BfffYi7Hnuev3nT+fza0pMmOiQzs1Hxmf4Q/qnxcW7bsov3/8aLeOvLlwxfwcxsknPSH8T3tj7D3965nd+58FQ+eNk5Ex2OmVlROOnncc/OvXzoXx/kFWfN45NvOt/34ptZyXDSH2DHsy+w5qtbOG3+dP757fVUVfgQmVnpcEbLcairh3d+aTNVFeV86Z0vZ/Z0T1xuZqXFd+/keLj5AE37O/jc2y5iybzpEx2OmVnR+Uw/R0tbJwBnn+xx8c2sNBWU9CWtkrRd0g5J1+fZf7qk/5b0oKRGSYtz9l0r6bH0cW0xgy+2ltYOABa6x62Zlahhk76kcuAW4HXAMuBqScsGFPs74KsRcT5wM/DXad15wEeBi4EVwEclzS1e+MXV0tZJbXUFM2vclm9mpamQM/0VwI6I2BkRXcA64KoBZZYB/50ub8rZfwWwMSL2RcR+YCOwavRhj42Wtg6f5ZtZSSvkQu6pwK6c9SaSM/dcDwBvAj4N/A4wU9L8QeoeN82UpDXAGoC6ujoaGxsLDP947e3tI67/aFMHMyo0qvcv1GjiHE+Os/imSqyOs/gmQ6yFJP18PZNiwPqfAv8o6Z3AT4BngJ4C6xIRtwK3AtTX10dDQ0MBYeXX2NjISOt/6Kc/ZMWZJ9PQcP6I379Qo4lzPDnO4psqsTrO4psMsRaS9JuA3IFnFgPNuQUiohl4I4CkWuBNEdEmqQloGFC3cRTxjpmunj6ebz/sYZPNrKQV0qa/GVgq6UxJVcBqYH1uAUkLJPW/1g3A2nT5TuBySXPTC7iXp9smnT0HOomARXOc9M2sdA2b9COiB7iOJFk/Anw7Ih6WdLOk16fFGoDtkh4F6oCPp3X3AR8j+eHYDNycbpt0dh9I7tE/Zfa0CY7EzGzsFNQjNyI2ABsGbLsxZ/l24PZB6q7l6Jn/pNWc3qO/yM07ZlbC3CM3tbut/0zfSd/MSpeTfqqlrZOZ7phlZiXOST/V0tbhs3wzK3lO+qmWtk4WzvFFXDMrbU76qZa2ThbO8pm+mZU2J32Odsxa6Hv0zazEOelztGOWB1szs1LnpM/RyVMWumOWmZU4J32SO3fAZ/pmVvqc9Mk50/fdO2ZW4pz0SXrjzqyuoLba88SbWWlz0icZd8d37phZFjjpk4yw6dE1zSwLnPSB5tZOj65pZpmQ+aTvGbPMLEsyn/T3pJOnLHLzjpllQEFJX9IqSdsl7ZB0fZ79p0naJOl+SQ9KujLdfoakDklb08fni/0BRqvF4+ibWYYMe4+ipHLgFuAykknSN0taHxHbcor9Bck0ip+TtIxklq0z0n2PR8QFxQ27ePo7ZnluXDPLgkLO9FcAOyJiZ0R0AeuAqwaUCWBWujwbaC5eiGPr6Jm+m3fMrPQVkvRPBXblrDel23LdBLxdUhPJWf77c/admTb7/FjSr40m2LHQ0trBzBp3zDKzbFBEDF1AegtwRUS8J12/BlgREe/PKfPB9LU+JekS4IvAS4FKoDYi9kpaDtwBnBsRBwa8xxpgDUBdXd3ydevWjfgDtbe3U1tbW3D5z/yikz2H+vj4q6aP+D1H4kTjnCiOs/imSqyOs/jGMtaVK1feFxH1wxaMiCEfwCXAnTnrNwA3DCjzMLAkZ30ncHKe12oE6od6v+XLl8dobNq06YTK/9Zn7op3fPHeUb3nSJxonBPFcRbfVInVcRbfWMYKbIlh8nlEFNS8sxlYKulMSVXAamD9gDJPA68BkPQSoAZ4TtJJ6YVgJJ0FLE1/ECaNlrZOj65pZpkxbEN2RPRIug64EygH1kbEw5JuJvllWQ/8H+BfJH2A5KLuOyMiJL0auFlSD9AL/GFE7BuzT3OCDvf0JjNm+SKumWVEQVcvI2IDyQXa3G035ixvAy7NU+87wHdGGeOYefbAYcDj6JtZdmS6R25zazp5iu/RN7OMyHTS332gf5pEJ30zy4ZMJ/3mVnfMMrNsyXTS393mjllmli2ZTvrNbZ0eXdPMMiXTSX93W6dH1zSzTMl00m9p6/DommaWKZlN+knHrC5OmeXmHTPLjswm/T1taccsn+mbWYZkNun3T57ie/TNLEsynPT7O2a5ecfMssNJ32f6ZpYhGU76HcyqqWCGO2aZWYZkOOl3umnHzDInw0m/w3fumFnmZDbp7/aMWWaWQZlM+v0ds9y8Y2ZZU1DSl7RK0nZJOyRdn2f/aZI2Sbpf0oOSrszZd0Nab7ukK4oZ/Ej1d8zyuDtmljXD3rqSTmx+C3AZ0ARslrQ+nSKx318A346Iz0laRjK14hnp8mrgXGAR8ENJ50REb7E/yIloTjtmeYRNM8uaQs70VwA7ImJnRHQB64CrBpQJYFa6PBtoTpevAtZFxOGIeALYkb7ehNrd1j95is/0zSxbFBFDF5DeDKyKiPek69cAF0fEdTllFgI/AOYCM4DXRsR9kv4RuCcivp6W+yLw/Yi4fcB7rAHWANTV1S1ft27diD9Qe3s7tbW1Q5b5j51d3P5oN59/7XRqKjTi9xqNQuKcDBxn8U2VWB1n8Y1lrCtXrrwvIuqHK1dIz6R8WXHgL8XVwJcj4lOSLgG+JumlBdYlIm4FbgWor6+PhoaGAsLKr7GxkeHq/6jtIWbVPMOq164c8fuMViFxTgaOs/imSqyOs/gmQ6yFJP0mYEnO+mKONt/0ezewCiAi7pZUAywosO64a27tZNEct+ebWfYU0qa/GVgq6UxJVSQXZtcPKPM08BoASS8BaoDn0nKrJVVLOhNYCvy8WMGP1O4DHW7PN7NMGvZMPyJ6JF0H3AmUA2sj4mFJNwNbImI98H+Af5H0AZLmm3dGcrHgYUnfBrYBPcD7JvrOHYCW1k7OO3XORIdhZjbuChptLCI2kNyGmbvtxpzlbcClg9T9OPDxUcRYVJ3dvew92OXeuGaWSZnrkbvngIdUNrPsylzS9+QpZpZlGUz66TSJHmHTzDIog0nfzTtmll3ZS/qtncyeVsn0Ks+YZWbZk72k73H0zSzDMpj03THLzLIrc0l/t+fGNbMMy1TSd8csM8u6TCV9d8wys6zLVNJvbnXHLDPLtkwl/d0H3DHLzLItU0n/6Jm+k76ZZVOmkv7uNnfMMrNsy1TSb2nr8Fm+mWVaxpK+e+OaWbYVlPQlrZK0XdIOSdfn2f8Pkramj0cltebs683ZN3CaxXHV0tbJKb5zx8wybNjGbUnlwC3AZSQTnW+WtD6dLQuAiPhATvn3AxfmvERHRFxQvJBHprO7l30Hu1jkM30zy7BCzvRXADsiYmdEdAHrgKuGKH818K1iBFdMu9MhlT3ujpllmZL5y4coIL0ZWBUR70nXrwEujojr8pQ9HbgHWNw/AbqkHmArycTon4iIO/LUWwOsAairq1u+bt26EX+g9vZ2amtrj9v+yN5ePrm5kz97eQ3L5peP+PWLZbA4JxvHWXxTJVbHWXxjGevKlSvvi4j6YQtGxJAP4C3AF3LWrwE+O0jZDw/cByxKn88CngTOHur9li9fHqOxadOmvNu/c9+uOP3D/xE7nn1hVK9fLIPFOdk4zuKbKrE6zuIby1iBLTFMPo+Igpp3moAlOeuLgeZByq5mQNNORDSnzzuBRo5t7x83njHLzKywNv3NwFJJZ0qqIknsx92FI+nFwFzg7pxtcyVVp8sLgEuBbQPrjoeWtg53zDKzzBs2A0ZEj6TrgDuBcmBtRDws6WaSPyf6fwCuBtalf2b0ewnwz5L6SH5gPhE5d/2Mp92+R9/MbPikDxARG4ANA7bdOGD9pjz1fgacN4r4iqa51UnfzCwzPXJ3H+hk4Rx3zDKzbMtE0u/vmLVwls/0zSzbMpH0+ztm+UzfzLIuE0m/uS2dPMVt+maWcZlI+rt9j76ZGZCRpH+0Y5abd8ws2zKS9DuYM72SaVUTP+aOmdlEykbSb+3kFN+5Y2aWkaTf1ski37ljZpaVpN/hcfTNzMhA0u/s7mX/oW7PmGVmRgaSfsuRGbPcvGNmloGkn3TM8pm+mVkWkn6r58Y1M+tX8kl/9wF3zDIz61fySb+51R2zzMz6lXzST2bM8lm+mRkUmPQlrZK0XdIOSdfn2f8Pkramj0cltebsu1bSY+nj2mIGX4hmT5NoZnbEsNMlSioHbgEuA5qAzZLW5851GxEfyCn/fuDCdHke8FGgHgjgvrTu/qJ+iiHsbuvgotPmjNfbmZlNaoWc6a8AdkTEzojoAtYBVw1R/mrgW+nyFcDGiNiXJvqNwKrRBHwiOrqSjlk+0zczSxQyMfqpwK6c9Sbg4nwFJZ0OnAn8aIi6p+aptwZYA1BXV0djY2MBYeXX3t5+pP7ug30AtLU8SWPjMyN+zbGQG+dk5jiLb6rE6jiLbzLEWkjSV55tMUjZ1cDtEdF7InUj4lbgVoD6+vpoaGgoIKz8Ghsb6a//sx3Pw133svLiC3nlixaM+DXHQm6ck5njLL6pEqvjLL7JEGshzTtNwJKc9cVA8yBlV3O0aedE6xZdi+fGNTM7RiFJfzOwVNKZkqpIEvv6gYUkvRiYC9yds/lO4HJJcyXNBS5Pt42L/iEYPJa+mVli2OadiOiRdB1Jsi4H1kbEw5JuBrZERP8PwNXAuoiInLr7JH2M5IcD4OaI2FfcjzC4lrZO5rpjlpnZEYW06RMRG4ANA7bdOGD9pkHqrgXWjjC+UWlp6/TommZmOUq6R25LW6dH1zQzy1HiSd8zZpmZ5SrZpN/R1UvroW7PjWtmlqNkk77v3DEzO17JJv3dR+7Rd9I3M+tXskm/uc2Tp5iZDVSySX932rzjwdbMzI4q2aTfnHbMqql0xywzs34lm/Q9Y5aZ2fFKNuk3t3a4acfMbICSTfq7D3T6zh0zswFKMun3d8xy846Z2bFKMum3+M4dM7O8SjTpJ/foe9wdM7NjlXTSX+TmHTOzY5Rm0m9Nx93xmb6Z2TEKSvqSVknaLmmHpOsHKfNWSdskPSzpmznbeyVtTR/HTbM4FloOdDJvRpU7ZpmZDTDszFmSyoFbgMtIJjrfLGl9RGzLKbMUuAG4NCL2Szo55yU6IuKCIsc9pJbWDo+uaWaWRyFn+iuAHRGxMyK6gHXAVQPCPNCOAAAI+ElEQVTK/D5wS0TsB4iIZ4sb5olpaetkke/RNzM7jnLmMc9fQHozsCoi3pOuXwNcHBHX5ZS5A3gUuJRk8vSbIuK/0n09wFagB/hERNyR5z3WAGsA6urqlq9bt27EH6i9vZ0P3ysuPqWCd5xbPeLXGWvt7e3U1tZOdBjDcpzFN1VidZzFN5axrly58r6IqB+uXCEToyvPtoG/FBXAUqABWAzcJemlEdEKnBYRzZLOAn4k6ZcR8fgxLxZxK3ArQH19fTQ0NBQQVn53/nATB7sPsXzZ2TQ0vGjErzPWGhsbGc3nHC+Os/imSqyOs/gmQ6yFNO80AUty1hcDzXnKfC8iuiPiCWA7yY8AEdGcPu8EGoELRxnzkPZ1Jr9H7phlZna8QpL+ZmCppDMlVQGrgYF34dwBrASQtAA4B9gpaa6k6pztlwLbGEP7D/cnfd+jb2Y20LDNOxHRI+k64E6S9vq1EfGwpJuBLRGxPt13uaRtQC/woYjYK+mVwD9L6iP5gflE7l0/Y2FvRx/gM30zs3wKadMnIjYAGwZsuzFnOYAPpo/cMj8Dzht9mIXrP9N3xywzs+OVXI/cfR3hjllmZoMovaR/ONy0Y2Y2iNJL+h19TvpmZoMouaS//3D4zh0zs0GUVNI/1NXDwW5fxDUzG0xJJf0j4+h73B0zs7xKK+m3pjNmzXLzjplZPqWV9NO5cX2mb2aWX4kl/eRMv85j6ZuZ5VVySX9mFe6YZWY2iBJL+h3Mqympj2RmVlQllSF3t3Uyrybf8P9mZgYllvSbWzuY66RvZjaokkn6Bw/3cKCzx2f6ZmZDKJmkf7inj99+2SLOmFUyH8nMrOhKJkPOm1HFZ6++kJcuKGiKADOzTCoo6UtaJWm7pB2Srh+kzFslbZP0sKRv5my/VtJj6ePaYgVuZmYnbtjTYknlwC3AZSQToG+WtD532kNJS4EbgEsjYr+kk9Pt84CPAvVAAPeldfcX/6OYmdlwCjnTXwHsiIidEdEFrAOuGlDm94Fb+pN5RDybbr8C2BgR+9J9G4FVxQndzMxOVCFJ/1RgV856U7ot1znAOZJ+KukeSatOoK6ZmY2TQq565rsHMvK8zlKgAVgM3CXppQXWRdIaYA1AXV0djY2NBYSVX3t7+6jqjxfHWVxTJU6YOrE6zuKbDLEWkvSbgCU564uB5jxl7omIbuAJSdtJfgSaSH4Icus2DnyDiLgVuBWgvr4+GhoaBhYpWGNjI6OpP14cZ3FNlThh6sTqOItvMsRaSPPOZmCppDMlVQGrgfUDytwBrASQtICkuWcncCdwuaS5kuYCl6fbzMxsAgx7ph8RPZKuI0nW5cDaiHhY0s3AlohYz9Hkvg3oBT4UEXsBJH2M5IcD4OaI2DcWH8TMzIaniOOa2CeUpOeAp0bxEguA54sUzlhynMU1VeKEqROr4yy+sYz19Ig4abhCky7pj5akLRFRP9FxDMdxFtdUiROmTqyOs/gmQ6wlMwyDmZkNz0nfzCxDSjHp3zrRARTIcRbXVIkTpk6sjrP4JjzWkmvTNzOzwZXimb6ZmQ3CSd/MLEOmZNIfbnx/SdWSbkv33yvpjPGPEiQtkbRJ0iPpPAP/O0+ZBkltkramjxsnKNYnJf0yjWFLnv2S9Jn0mD4o6aIJiPHFOcdpq6QDkv5kQJkJO56S1kp6VtJDOdvmSdqYziexMe2Znq/uuM07MUicfyvpV+m/7XclzRmk7pDfk3GI8yZJz+T8+145SN1h5wAZh1hvy4nzSUlbB6k7bscUgIiYUg+SXsGPA2cBVcADwLIBZd4LfD5dXg3cNkGxLgQuSpdnAo/mibUB+I9JcFyfBBYMsf9K4Pskg+i9Arh3EnwPdpN0SJkUxxN4NXAR8FDOtr8Brk+Xrwc+mafePJJhS+YBc9PlueMc5+VARbr8yXxxFvI9GYc4bwL+tIDvxpA5YjxiHbD/U8CNE31MI2JKnukXMr7/VcBX0uXbgddIGvcZ0yOiJSJ+kS6/ADzC1B1a+irgq5G4B5gjaeEExvMa4PGIGE3v7aKKiJ8AA4cZyf0ufgV4Q56q4zrvRL44I+IHEdGTrt5DMjjihBrkeBaikBxRVEPFmuaetwLfGssYCjUVk34hY/QfKZN+kduA+eMS3SDSJqYLgXvz7L5E0gOSvi/p3HEN7KgAfiDpvnSo64Em29wIqxn8P9FkOJ796iKiBZKTAODkPGUm27H9PZK/6vIZ7nsyHq5Lm6HWDtJcNtmO568BeyLisUH2j+sxnYpJv5Ax+gsax3+8SKoFvgP8SUQcGLD7FyRNFC8DPksyYulEuDQiLgJeB7xP0qsH7J80x1TJaK+vB/41z+7JcjxPxGQ6tn8O9ADfGKTIcN+TsfY54GzgAqCFpNlkoElzPFNXM/RZ/rge06mY9Asd338JgKQKYDYj+zNx1CRVkiT8b0TEvw3cHxEHIqI9Xd4AVCoZnnpcRURz+vws8F2SP5FzFXLcx8vrgF9ExJ6BOybL8cyxp78ZLH1+Nk+ZSXFs0wvIvwW8LdLG5oEK+J6MqYjYExG9EdEH/Msg7z8pjiccyT9vBG4brMx4H9OpmPQLGd9/PdB/B8SbgR8N9iUeS2lb3heBRyLi7wcpc0r/9QZJK0j+TfaOX5QgaYakmf3LJBf1HhpQbD3wjvQunlcAbf3NFhNg0DOnyXA8B8j9Ll4LfC9PmQmfd0LJFKcfBl4fEYcGKVPI92RMDbiO9DuDvH8hOWK8vBb4VUQ05ds5Icd0vK4YF/NBcifJoyRX6P883XYzyRcWoIbkT/8dwM+BsyYozleR/Fn5ILA1fVwJ/CHwh2mZ64CHSe4wuAd45QTEeVb6/g+ksfQf09w4BdySHvNfAvUTdEynkyTx2TnbJsXxJPkhagG6Sc42301yLem/gcfS53lp2XrgCzl1fy/9vu4A3jUBce4gaQfv/5723/22CNgw1PdknOP8Wvr9e5AkkS8cGGe6flyOGO9Y0+1f7v9u5pSdsGMaER6GwcwsS6Zi846ZmY2Qk76ZWYY46ZuZZYiTvplZhjjpm5lliJO+mVmGOOmbmWXI/wdSgGITODkaQQAAAABJRU5ErkJggg==
" style="border: 0px; box-sizing: border-box; height: auto; max-width: 100%; vertical-align: middle;" /></div>
</div>
<div class="output_area" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row; padding: 0px;">
<div class="prompt" style="-webkit-user-select: none; box-sizing: border-box; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0px 0.4em; text-align: right;">
</div>
<div class="output_png output_subarea " style="-webkit-box-flex: 1; box-sizing: border-box; flex: 1 1 0%; max-width: calc(100% - 14ex); overflow-x: auto; padding: 0.4em;">
<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZsAAAEWCAYAAACwtjr+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl8XOV97/HPV5J3yfKGBUhml0PBBgIOS7NcJ+Sy3aakWW5JucVJaLhpSZs0TRPS3hto2rRJm5SGhJKQQIA2JUBILm7DmoBCFsxObMCAzWrjDbCxLRsvkn73j/OMPZJHtixrliN936/XoJnnPOec3xzk+ek85zfPUURgZmZWTnXVDsDMzIY/JxszMys7JxszMys7JxszMys7JxszMys7JxszMys7JxurSZLOlXRnBfZziKSQ1FDufQ2EpGsk/V214zAbak42VjWS3ibp15LWS1or6VeS3gIQEd+PiNOqHePekHSKpE2Smkose1TSJ6oRV9r/aElfk7RcUqek5yVdWuZ9fljSL8u5D8sPJxurCkkTgf8CvgFMAVqBvwG2VjOuvdH3bCgi7gOWA+/v028WcBRwfeWi28XngTnAiUAT8E7g0SrGA4Ck+mrHYJXhZGPVMhMgIq6PiO6IeCMi7oyIhbDrX8VpqOvjkpZIWifpcklKy+rTX+2vpr/YP1E8NCbpBUnvLtrWJZL+vVRQkj4iabGkjZKek/S/i5bNTWcGn5O0CvheiU1cC5zXp+084CcR8Vrazk2SVqUzunslHd1PLLucGaT3dUR6PkbSVyW9JGm1pG9JGlfyaMNbgB9HxIrIvBAR1xVt9wVJn5f0ZDq+35M0tmj570h6TNLr6Wz0mKJlMyT9SNIrkl6T9E1JvwV8CzglnUm9nvpeI+kKSbdK2gS8U1KHpD/q732n9/wn6f/9Rkl/K+lwSfdJ2iDpRkmj+3nfViOcbKxangG6JV0r6UxJkwewzu+QfWgeC/xP4PTU/jHgTOA44HjgvfsQ15q0n4nAR4BLJR1ftHx/sjOxg4ELSqz/b8DbJR0EIKkO+APguqI+twHtwHTgEeD7g4z1K2RJ+zjgCLKzwy/003cB8On0oT27kKj7OJfsmB6etvt/0ns4Hrga+N/AVODbwPyU7OrJzlBfBA5JMfwgIhYDHwfui4jGiJhUtJ8/AL5EdoY10GG2M4ATgJOBzwJXpnhnALOADw1wO1YlTjZWFRGxAXgbEMB3gFckzZfUspvVvhwRr0fES8A9ZB+ykCWer0fE8ohYB3x5H+L6SUQ8m/76/zlwJ/D2oi49wMURsTUi3iix/jLg58D/Sk2nAmOBnxT1uToiNkbEVuAS4FhJzXsTZ0oWHwP+PCLWRsRG4O+Bc/pZ5R/IktO5wEPAy5Lm9enzzYhYFhFryZJB4QP8Y8C3I+L+dBZ6Ldlw58lkw3IHAn8ZEZsiYktE7CmB3BIRv4qInojYMsC3/JWI2BARTwCPA3dGxHMRsZ4seb95gNuxKnGysaqJiMUR8eGIaCP76/RA4F92s8qqouebgcb0/EBgWdGy4ud7JZ1lLUgFC68DZwHTirq8MoAPyOKhtD8E/iMitqft10v6sqRnJW0AXkj9pu26md3aDxgPPJyGtl4Hbk/tu0hJ4vKIeCswiSyZXJ2GuwqKj9uLZMcVsrO4vyjsJ+1rRlo+A3gxIrr2IvbB/P9ZXfT8jRKvG7Ga5mRjNSEingKuIUs6e2sl0Fb0ekaf5ZvIPpgL9i+1EUljgJuBrwItaejnVqB4yGkg06T/CGiV9E7gffQeQvsD4Gzg3UAz2dATffZRMm5JxXG/SvYhe3RETEqP5ojY44duuj52ObCOrHChoPi4HQSsSM+XAV8q2s+kiBgfEdenZQf1LZYo7Kq/EHb3Punn/4/lm5ONVYWkIyX9haS29HoG2bDNgkFs7kbgk5JaJU0CPtdn+WPAOZJGSZoDfKCf7YwGxgCvAF2SzgT2uvw6IjYBPyQrIHgxIh4qWtxENgT1GtkH7N/vZlO/AY6WdFy6WH9J0T56yIYfL5U0HSC9/9NLbUjSp1KBwzhJDWkIrYneFWkXSmqTNAX4K+CG1P4d4OOSTlJmgqT/oazE+wGyZP/l1D5W0lvTequBtgFcvH8MeJ+k8an44fw99LcccrKxatkInATcn6qSFpCNxf/FILb1HbJrKwvJPjxvBbqA7rT8/5Jd9F5HVl79H6U2kq57/BlZ8lpHdhYyfxDxQDaUdjC9z2pIr18EXgaeZDfJNSKeAb4I/BRYwq4X0z8HLAUWpCG5nwJv6mdzbwBfIxuKfBW4EHh/RDxX1Oc/yI7jc+nxdymOh8iu23yT7LgsBT6clnUD7yErUHiJrPT799P27gaeAFZJerW/9wlcCmwjS07XMviCCath8s3TbLhJZyTfioiDqx1LXkh6AfijiPhptWOx4clnNpZ7aWjorDQ81ApcDPy42nGZ2U5ONjYciGx4bB3ZMNpi+v++iZlVgYfRzMys7HxmY2ZmZVcT06rXgmnTpsUhhxwyqHU3bdrEhAkThjagMshLnJCfWB3n0MpLnJCfWMsd58MPP/xqRJT8MnEvEeFHBCeccEIM1j333DPodSspL3FG5CdWxzm08hJnRH5iLXecwEMxgM9YD6OZmVnZOdmYmVnZOdmYmVnZOdmYmVnZOdmYmVnZOdmYmVnZOdmYmVnZlS3ZSLpa0hpJjxe1TZF0l6Ql6efk1C5Jl0laKmlh8T3fJc1L/ZcU38ZW0gmSFqV1LivcU72/fZTLLY+9zN0vbS/nLszMcq+cZzbXAGf0absI+FlEtAM/S68BzgTa0+MC4ArIEgfZDL4nkd3r/OKi5HFF6ltY74w97KMsbl20kjtfcLIxM9udsiWbiLgXWNun+WyymyORfr63qP269IXUBcAkSQcApwN3RcTaiFgH3AWckZZNjIj70jdYr+uzrVL7KIuZLU2s3hxs2d69585mZiNUpedGa4mIlQARsbJwO1uglexe5gXLU9vu2peXaN/dPnYh6QKysyNaWlro6OjY6ze0/bUuArjxtg4Omli/1+tXUmdn56DeYzXkJVbHObTyEifkJ9ZaibNWJuJUibYYRPteiYgrgSsB5syZE3Pnzt3bTbD/qg186ze/oPmgI5l7XOueV6iijo4OBvMeqyEvsTrOoZWXOCE/sdZKnJWuRludhsBIP9ek9uXAjKJ+bcCKPbS3lWjf3T7K4tBpE6gTLFndWc7dmJnlWqWTzXygUFE2D7ilqP28VJV2MrA+DYXdAZwmaXIqDDgNuCMt2yjp5FSFdl6fbZXaR1mMaainZbx4ZvXGcu7GzCzXyjaMJul6YC4wTdJysqqyLwM3SjofeAn4YOp+K3AWsBTYDHwEICLWSvpb4MHU74sRUSg6+GOyirdxwG3pwW72UTYHNtaxZI3PbMzM+lO2ZBMRH+pn0akl+gZwYT/buRq4ukT7Q8CsEu2vldpHObU21vHoc5vYsr2bsaNqu0jAzKwaPIPAEGhtrKMn4NlXfHZjZlaKk80QaG3MDuNSD6WZmZXkZDME9p8gGupcJGBm1h8nmyHQUCcOmTaBZ1z+bGZWkpPNEGmf3sgSn9mYmZXkZDNE2luaeHHtZs+RZmZWgpPNEJnZ0kiEiwTMzEpxshkiM1uaACcbM7NSnGyGyCFTJ7gizcysH042Q2R0Q50r0szM+uFkM4RmtjSyZI3PbMzM+nKyGULt05t4ae1m3tjmijQzs2JONkNoZksT4TnSzMx24WQzhGa2NAJ4KM3MrA8nmyF0yLRCRZrPbMzMijnZDKFR9XUcOm2Cp60xM+vDyWaIzWxp8pmNmVkfTjZDrL2lkWXrXJFmZlbMyWaIuSLNzGxXTjZDrFCR5mlrzMx2crIZYgdPncCoelekmZkVc7IZYq5IMzPblZNNGbS3NPGMv9hpZraDk00ZzJzexLK1b7B5W1e1QzEzqwlONmVQKBJ4ds2mKkdiZlYbnGzKoD3dtdMVaWZmGSebMjhk6visIs3XbczMACebsmior+OwaY0scfmzmRngZFM27S2NHkYzM0ucbMpkZksTy9e9waatrkgzM6tKspH055KekPS4pOsljZV0qKT7JS2RdIOk0anvmPR6aVp+SNF2Pp/an5Z0elH7GaltqaSLKv8OiyrSPEeamVnlk42kVuDPgDkRMQuoB84BvgJcGhHtwDrg/LTK+cC6iDgCuDT1Q9JRab2jgTOAf5VUL6keuBw4EzgK+FDqW1E7K9KcbMzMqjWM1gCMk9QAjAdWAu8CfpiWXwu8Nz0/O70mLT9VklL7DyJia0Q8DywFTkyPpRHxXERsA36Q+lbUwVPGM7q+ztPWmJmRfehXVES8LOmrwEvAG8CdwMPA6xFRuMCxHGhNz1uBZWndLknrgampfUHRpovXWdan/aRSsUi6ALgAoKWlhY6OjkG9p87OzpLrTh8X3Lf4RTrGrx7Udodaf3HWorzE6jiHVl7ihPzEWitxVjzZSJpMdqZxKPA6cBPZkFdfUViln2X9tZc6W4sSbUTElcCVAHPmzIm5c+fuLvR+dXR0UGrdN698lEdeXFdyWTX0F2ctykusjnNo5SVOyE+stRJnNYbR3g08HxGvRMR24EfAbwOT0rAaQBuwIj1fDswASMubgbXF7X3W6a+94mZOb+Tl112RZmZWjWTzEnCypPHp2supwJPAPcAHUp95wC3p+fz0mrT87oiI1H5OqlY7FGgHHgAeBNpTddtosiKC+RV4X7soFAksXeMiATMb2SqebCLifrIL/Y8Ai1IMVwKfAz4taSnZNZmr0ipXAVNT+6eBi9J2ngBuJEtUtwMXRkR3uu7zCeAOYDFwY+pbcb5rp5lZpuLXbAAi4mLg4j7Nz5FVkvXtuwX4YD/b+RLwpRLttwK37nuk++agKeMZ3VDHEp/ZmNkI5xkEyiibI22Cz2zMbMRzsimzmS1NnpDTzEY8J5sym9mSVaR1uiLNzEYwJ5syc0WamZmTTdnN9F07zcycbMptR0Wak42ZjWBONmVWXycO36/Rsz+b2YjmZFMBM1safWZjZiOak00FzGxpYsX6LWzcsr3aoZiZVYWTTQW0T8+mrXFFmpmNVE42FVCoSPOXO81spHKyqYAZU8YzpqHO5c9mNmI52VTAjoo0D6OZ2QjlZFMhrkgzs5HMyaZC2luaWOmKNDMboZxsKmRHkYCH0sxsBHKyqZDCXTs9lGZmI5GTTYW0TS5UpPnMxsxGHiebCqmvE0dMb3T5s5mNSE42FeS7dprZSOVkU0HtLY2s2rCFDa5IM7MRxsmmgmZO97Q1ZjYyOdlU0M450nzdxsxGFiebCmqbPI6xo1yRZmYjj5NNBdWlirQla3xmY2Yji5NNhc2c3uTyZzMbcZxsKqy9pYnVG7ay/g1XpJnZyOFkU2GFaWuWeijNzEYQJ5sKK1SkuUjAzEYSJ5sKa500jnGj6n3dxsxGlKokG0mTJP1Q0lOSFks6RdIUSXdJWpJ+Tk59JekySUslLZR0fNF25qX+SyTNK2o/QdKitM5lklSN91nKjoo0n9mY2QhSrTObrwO3R8SRwLHAYuAi4GcR0Q78LL0GOBNoT48LgCsAJE0BLgZOAk4ELi4kqNTngqL1zqjAexqw9hZPyGlmI0vFk42kicA7gKsAImJbRLwOnA1cm7pdC7w3PT8buC4yC4BJkg4ATgfuioi1EbEOuAs4Iy2bGBH3RUQA1xVtqybMbGlizcatrN/sijQzGxkaqrDPw4BXgO9JOhZ4GPgk0BIRKwEiYqWk6al/K7CsaP3lqW137ctLtO9C0gVkZ0C0tLTQ0dExqDfU2dm5V+tuWdMFwI133Ev75PpB7XMw9jbOaspLrI5zaOUlTshPrLUSZzWSTQNwPPCnEXG/pK+zc8islFLXW2IQ7bs2RlwJXAkwZ86cmDt37m7C6F9HRwd7s+7hazfzL4/cw4QD25l70kGD2udg7G2c1ZSXWB3n0MpLnJCfWGslzmpcs1kOLI+I+9PrH5Iln9VpCIz0c01R/xlF67cBK/bQ3laivWa4Is3MRpqKJ5uIWAUsk/Sm1HQq8CQwHyhUlM0DbknP5wPnpaq0k4H1abjtDuA0SZNTYcBpwB1p2UZJJ6cqtPOKtlUT6upEe4vnSDOzkWNAw2iSbgauBm6LiJ4h2O+fAt+XNBp4DvgIWeK7UdL5wEvAB1PfW4GzgKXA5tSXiFgr6W+BB1O/L0bE2vT8j4FrgHHAbelRU9qnN3HvkleqHYaZWUUM9JrNFWQf8pdJugm4JiKeGuxOI+IxYE6JRaeW6BvAhf1s52qyJNi3/SFg1mDjq4SZLY3c/MhyXt+8jUnjR1c7HDOzshrQMFpE/DQiziW7tvICcJekX0v6iKRR5QxwuNpxI7U1/nKnmQ1/A75mI2kq8GHgj4BHyb6YeTzZ91tsLx0xPZuQ00UCZjYSDPSazY+AI4F/A95T+D4McIOkh8oV3HDWOmkc40fXe9oaMxsRBnrN5psRcXepBRFR6tqL7UFdnWif7mlrzGxkGGiymSTpfX3a1gOLImJNqRVsz9pbmuh42hVpZjb8DfSazfnAd4Fz0+M7wKeBX0n6wzLFNuzNbGnk1c6trNu0rdqhmJmV1UCTTQ/wWxHx/oh4P3AUsJVsxuXPlSu44a7dFWlmNkIMNNkcEhGri16vAWamL1F66uJBandFmpmNEAO9ZvMLSf8F3JRevx+4V9IE4PWyRDYCtE4ax4TR9SxxsjGzYW6gyeZC4H3A28hmVb4OuDl9u/+dZYpt2JPEES1NPOPyZzMb5vaYbCTVk01w+W7g5vKHNLLMnN7IPa5IM7Nhbo/XbCKiG9gsqbkC8Yw4M1uaXJFmZsPeQIfRtgCLJN0FbCo0RsSflSWqEaS9ZWeRwEmHTa1yNGZm5THQZPOT9LAhVih/fmZNp5ONmQ1bA0o2EXGtpHHAQRHxdJljGlEObB5L45gGV6SZ2bA2oO/ZSHoP8Bhwe3p9nKT55QxspJDEzJZGnlrpZGNmw9dAv9R5CXAi6Ts16eZnh5YpphFnVmszT6xYT09PVDsUM7OyGGiy6YqI9X3a/Mk4RGa1NrNpWzfPvbppz53NzHJooMnmcUl/ANRLapf0DeDXZYxrRDmmLasqf/zlvvnczGx4GGiy+VPgaLLJN68HNgCfKldQI80R+zUydlQdC5c72ZjZ8DTQarTNwF+nhw2xhvo6jjpgos9szGzYGuhtoWcCnwEOKV4nIt5VnrBGntmtzdz08HK6e4L6OlU7HDOzITXQL3XeBHyL7AZq3eULZ+Sa1drMtfe9yPOvdnLE9KZqh2NmNqQGmmy6IuKKskYywh3TNgmARS+vd7Ixs2FnoAUC/ynpTyQdIGlK4VHWyEaYw/eb4CIBMxu2BnpmMy/9/MuitgAOG9pwRi4XCZjZcDbQajTPFlABx7RN4saHlrlIwMyGnd0Oo0n6bNHzD/ZZ9vflCmqkmtXazOZt3Tz3iu/caWbDy56u2ZxT9PzzfZadMcSxjHiFmQQWeSjNzIaZPSUb9fO81GvbR4fv18i4UfUuEjCzYWdPySb6eV7q9V6RVC/pUUn/lV4fKul+SUsk3SBpdGofk14vTcsPKdrG51P705JOL2o/I7UtlXTRvsRZSfV14qgDXSRgZsPPnpLNsZI2SNoIHJOeF17P3sd9fxJYXPT6K8ClEdEOrAPOT+3nA+si4gjg0tQPSUeRDfMdTTak968pgdUDlwNnAkcBH0p9c2F2azNPrNhAt283YGbDyG6TTUTUR8TEiGiKiIb0vPB61GB3KqkN+B9kMxIgScC7gB+mLtcC703Pz06vSctPTf3PBn4QEVsj4nlgKdk9d04ElkbEcxGxDfhB6psLs1ubeWN7N8+6SMDMhpGBfqlzqP0L8FmgJ72eCrweEV3p9XKgNT1vBZYBpOXrU/8d7X3W6a89F2YXigR83cbMhpGBfqlzyEj6HWBNRDwsaW6huUTX2MOy/tpLJdCSY1KSLgAuAGhpaaGjo6P/wHejs7Nz0Ov21RPB6Hq4/YEnmLpx6ZBss2Ao4yy3vMTqOIdWXuKE/MRaK3FWPNkAbwV+V9JZwFhgItmZziRJDenspQ1YkfovB2YAyyU1AM3A2qL2guJ1+mvvJSKuBK4EmDNnTsydO3dQb6ijo4PBrlvKMU/9mrXA3Lm/PWTbhKGPs5zyEqvjHFp5iRPyE2utxFnxYbSI+HxEtEXEIWQX+O+OiHOBe4APpG7zgFvS8/nsnC7nA6l/pPZzUrXaoUA78ADwINCeqttGp33Mr8BbGzKz25p5csUGurp79tzZzCwHqnXNppTPAZ+WtJTsmsxVqf0qYGpq/zRwEUBEPAHcCDwJ3A5cGBHd6czoE8AdZNVuN6a+ubGzSGBTtUMxMxsS1RhG2yEiOoCO9Pw5skqyvn22AB/s256WfQn4Uon2W4FbhzDUiprdunMmgTft79sNmFn+1dKZjSWH7dfI+NH1/nKnmQ0bTjY1qL5OHH3gRBYuf73aoZiZDQknmxo1u3UST650kYCZDQ9ONjVqdttEtmzvcZGAmQ0LTjY1qlAk4KE0MxsOnGxq1KHTGpngIgEzGyacbGpUViTQzEInGzMbBpxsatis1mYWu0jAzIYBJ5sadkxbM1u297DUtxsws5xzsqlhs3YUCXgozczyzcmmhh02bYKLBMxsWHCyqWF1deLo1mYWOdmYWc452dS42a2+3YCZ5Z+TTY07pq2ZrV09LFnjIgEzyy8nmxpXKBJY5CIBM8sxJ5sad+jUCTSOafB1GzPLNSebGleXbjfgZGNmeeZkkwOzW5t5cuUGtrtIwMxyyskmB2a3NbOtq4clq10kYGb55GSTA4XbDfjLnWaWV042OXBIKhJY+LLvbWNm+eRkkwN1dWJW60QWvbyh2qGYmQ2Kk01OzE63G3CRgJnlkZNNTsxum8S2rh6eWb2x2qGYme01J5uccJGAmeWZk01OHDxlPE1jGnxvGzPLJSebnMiKBJp9ZmNmueRkkyOz25pZvGoj27pcJGBm+eJkkyOzW5tdJGBmueRkkyMuEjCzvHKyyZGDp46naWwDC51szCxnKp5sJM2QdI+kxZKekPTJ1D5F0l2SlqSfk1O7JF0maamkhZKOL9rWvNR/iaR5Re0nSFqU1rlMkir9PstBErNdJGBmOVSNM5su4C8i4reAk4ELJR0FXAT8LCLagZ+l1wBnAu3pcQFwBWTJCbgYOAk4Ebi4kKBSnwuK1jujAu+rIma3NvPUShcJmFm+VDzZRMTKiHgkPd8ILAZagbOBa1O3a4H3pudnA9dFZgEwSdIBwOnAXRGxNiLWAXcBZ6RlEyPivogI4LqibeXerNZmtnW7SMDM8qWhmjuXdAjwZuB+oCUiVkKWkCRNT91agWVFqy1PbbtrX16ivdT+LyA7A6KlpYWOjo5BvY/Ozs5Br7u33ticndH88O4HeHXGqL1at5Jx7qu8xOo4h1Ze4oT8xForcVYt2UhqBG4GPhURG3ZzWaXUghhE+66NEVcCVwLMmTMn5s6du4eoS+vo6GCw6+6tiODvHriTbY37M3fu7L1at5Jx7qu8xOo4h1Ze4oT8xForcValGk3SKLJE8/2I+FFqXp2GwEg/16T25cCMotXbgBV7aG8r0T4sSGJ2WzOLPG2NmeVINarRBFwFLI6Ify5aNB8oVJTNA24paj8vVaWdDKxPw213AKdJmpwKA04D7kjLNko6Oe3rvKJtDQuzWpt5atUGtnZ1VzsUM7MBqcYw2luBPwQWSXostf0V8GXgRknnAy8BH0zLbgXOApYCm4GPAETEWkl/CzyY+n0xItam538MXAOMA25Lj2Fjdmsz27uDZ1Z1MrutudrhmJntUcWTTUT8ktLXVQBOLdE/gAv72dbVwNUl2h8CZu1DmDXtmNZJACx6eb2TjZnlgmcQyKEZU8bRPG4Ui/zlTjPLCSebHJLErNaJLHr59WqHYmY2IE42OTW7dRJPr9roIgEzywUnm5wqLhIwM6t1TjY5dUwqDFjooTQzywEnm5xqm5wVCXgGaDPLAyebnCrcbmChZxIwsxxwssmx2W3NPLPaRQJmVvucbHKsUCTw9CrfbsDMapuTTY7Nbk1FAh5KM7Ma52STY22TxzFpvIsEzKz2OdnkmIsEzCwvnGxybnZrViSwZbuLBMysdjnZ5Nzs1ma6elwkYGa1zckm52YVigR83cbMapiTTc61TR7H5PGjeNzXbcyshjnZ5Fx2u4Fm39vGzGqak80wcEybiwTMrLY52QwDhSKBp1wkYGY1yslmGCgUCSxa7tsNmFltcrIZBlonjWP/iWP5pzue5su3PcWq9VuqHZKZWS9ONsOAJK756Ft4e/t+XHnvs7z9H+/m0zc+xuKVG6odmpkZAA3VDsCGxpH7T+Tyc49n2drNXPXL57nxoWX86JGXeXv7NP7o7YfxjvZp1Q7RzEYwJ5thZsaU8Vzyu0fz5++eyfcfeJFrfvUC865+gDe1NPG2/bZzSlc3Yxrqqx2mmY0wHkYbpprHj+JP5h7BLz/3Lr72wWOR4KrHt/G2r9zD5fcs5fXN26odopmNIE42w9zohjref0Ibt33y7XxmzliO3L+Jf7rjaU75h7u5+JbHefG1TdUO0cxGAA+jjRCSmDWtnk984CSeWrWB7/7ief7jgZe4bsGLnH7U/nzsHYdxwsGTqx2mmQ1TTjYj0JH7T+SrHzyWvzz9TVz76xf49wUvcvsTqzj+oEmccvhUpk4Yw9TG0UxrzH5OnTCGyeNH0VDvE2EzGxwnmxGsZeJYPnvGkVz4ziO46aFlXLfgRb718+fo7old+kowadwopjaOYeqE3okoS0yjmZKeTxw7iqaxDYwd5UIEM8s42RgTxjTw4bceyoffeig9PcH6N7bz2qatvNq5jbWbtvFaZ/b8tU1bea1zG691bmPxqg281rmN9W9s73e7o+vraBrbQNPYBhrHNtA0ZlR6PWpHe+/Xo2gc08DEsQ2s29JD59YuJoyuR1IFj4aZlcOwTTaSzgC+DtQD342IL1c5pFyoqxOTJ4xm8oTRHDF9z/23d/ewbtO2Xslow5btbNzSlR7bd/zs3NrFS2s3s3FLFxvS69j1JGo7QKCMAAANyklEQVSnjjuoU5YMJ6ZE1FhIXmOKktSO9lGpPVteXyfqpPQze291EvUSdXXsWCaRtUmpDzvWbagr9HHCM9sXwzLZSKoHLgf+O7AceFDS/Ih4srqRDT+j6uuYPnEs0yeO3et1e3qCzdu7eyWkQpJ6eOETHHjwYXRu6WLDli46t+5MWGs3beOl1zan9u1s2d5ThnfWW0OdaKgXo+rqqK8XDXV1NNSJru1baXqog/q6LDGNqq+jvk6Mqlf6mb2uT4mt8GioyxJbQ6+2uizB1fdOdIVkqUJCVJYoVZQU+1telxLr4hVdrH/sZSDrJ7KhUaH0M3tNr9dF/VJfSi0rrLPL9vrfdt2OZb3XXbaxh6dXbUzvJ1un8N6K12NHXDvjqEvPKRVHWpcd+2LHsUir7NhmirTX61LLunqC7d09vfZRHJf1NiyTDXAisDQingOQ9APgbMDJpobU1Sk7WxnTwAHNvZc1rXuGue84fEDb2dbVw6at6Uxqa5awNm3torsn6ImgJyh6HnT3kD3vScsiiIjUh9QedPVkbV3dPXT1ZK+7uoOunh62dwfdPT0sX7GKafs1F7VlH0DdqW9nVxc9PTu31d0TdEfsWF7Yzy59erL9lLh8NngLHxvCjZXRr+6tdgQDd+dte+zSN7n1SkrsXLgzKe9Mljvbd65Hn+0VJ+NCgq9L264TbN26lbEL7qaubmdb7z8kxNXz3sJBU8fv27HYg+GabFqBZUWvlwMn9e0k6QLgAoCWlhY6OjoGtbPOzs5Br1tJeYkT9j3W+vTYJ9rzhjrHbqexcTA3rhM7PzL6FylZBukR2aMHsvbY2d5D9H5d9Lxz82bGjxtPIXdF+k9hu4V97Xy+s98ufYNe2yn07Slap7CNSC1RYp2+/SLgjS1bGDN2LKT3WLzfQnz9xdY3huJ+vbfTO84dx5reC/oeq75t27ZtY/To0b2Ggou33yvGPnH2Pj59tl20w1LHu/f6qV/h2BRttxDHtjE9NDRs3+UYFh+zhx9cwHNjy1ttOlyTTal/xbv8jRgRVwJXAsyZMyfmzp07qJ11dHQw2HUrKS9xQn5idZxDKy9xQn5irZU4h+sXJ5YDM4petwErqhSLmdmIN1yTzYNAu6RDJY0GzgHmVzkmM7MRa1gOo0VEl6RPAHeQjbhfHRFPVDksM7MRa1gmG4CIuBW4tdpxmJnZ8B1GMzOzGuJkY2ZmZedkY2ZmZedkY2ZmZafY7UyII4ekV4AXB7n6NODVIQynXPISJ+QnVsc5tPISJ+Qn1nLHeXBE7LenTk42Q0DSQxExp9px7Ele4oT8xOo4h1Ze4oT8xForcXoYzczMys7JxszMys7JZmhcWe0ABigvcUJ+YnWcQysvcUJ+Yq2JOH3NxszMys5nNmZmVnZONmZmVnZONntB0hmSnpa0VNJFJZaPkXRDWn6/pEOqEOMMSfdIWizpCUmfLNFnrqT1kh5Ljy9UOs4UxwuSFqUYHiqxXJIuS8dzoaTjqxTnm4qO1WOSNkj6VJ8+VTmmkq6WtEbS40VtUyTdJWlJ+jm5n3XnpT5LJM2rQpz/JOmp9P/2x5Im9bPubn9PKhTrJZJeLvr/e1Y/6+72M6ICcd5QFOMLkkreC7zSxxRItwn1Y48PslsVPAscBowGfgMc1afPnwDfSs/PAW6oQpwHAMen503AMyXinAv8Vw0c0xeAabtZfhZwG9mdV08G7q+BmOuBVWRfZKv6MQXeARwPPF7U9o/ARen5RcBXSqw3BXgu/Zycnk+ucJynAQ3p+VdKxTmQ35MKxXoJ8JkB/G7s9jOi3HH2Wf414Au1cEwjwmc2e+FEYGlEPBcR24AfAGf36XM2cG16/kPgVEl7vtH8EIqIlRHxSHq+EVgMtFYyhiF0NnBdZBYAkyQdUOWYTgWejYjBzjYxpCLiXmBtn+bi38NrgfeWWPV04K6IWBsR64C7gDMqGWdE3BkRXenlArI76lZdP8d0IAbyGTFkdhdn+tz5n8D15dr/3nKyGbhWYFnR6+Xs+iG+o0/6R7QemFqR6EpIw3hvBu4vsfgUSb+RdJukoysa2E4B3CnpYUkXlFg+kGNeaefQ/z/gWjimAC0RsRKyPz6A6SX61Nqx/SjZWWwpe/o9qZRPpCG/q/sZmqylY/p2YHVELOlnecWPqZPNwJU6Q+lbNz6QPhUhqRG4GfhURGzos/gRsmGgY4FvAP+v0vElb42I44EzgQslvaPP8po5ngDpFuO/C9xUYnGtHNOBqpljK+mvgS7g+/102dPvSSVcARwOHAesJBui6qtmjinwIXZ/VlPxY+pkM3DLgRlFr9uAFf31kdQANDO40/F9ImkUWaL5fkT8qO/yiNgQEZ3p+a3AKEnTKhwmEbEi/VwD/JhsGKLYQI55JZ0JPBIRq/suqJVjmqwuDDemn2tK9KmJY5sKE34HODfSxYS+BvB7UnYRsToiuiOiB/hOPzHUyjFtAN4H3NBfn2ocUyebgXsQaJd0aPoL9xxgfp8+84FCVc8HgLv7+wdULmms9ipgcUT8cz999i9cS5J0ItnvwWuVixIkTZDUVHhOdrH48T7d5gPnpaq0k4H1heGhKun3r8VaOKZFin8P5wG3lOhzB3CapMlpSOi01FYxks4APgf8bkRs7qfPQH5Pyq7PtcLf6yeGgXxGVMK7gaciYnmphVU7ppWsRsj7g6w66hmyipO/Tm1fJPvHAjCWbIhlKfAAcFgVYnwb2an7QuCx9DgL+Djw8dTnE8ATZNUyC4DfrkKch6X9/ybFUjiexXEKuDwd70XAnCr+vx9Pljyai9qqfkzJkt9KYDvZX9bnk10n/BmwJP2ckvrOAb5btO5H0+/qUuAjVYhzKdk1jsLvaaGS80Dg1t39nlQh1n9Lv4MLyRLIAX1jTa93+YyoZJyp/ZrC72VR36oe04jwdDVmZlZ+HkYzM7Oyc7IxM7Oyc7IxM7Oyc7IxM7Oyc7IxM7Oyc7Kx3JHUJumWNFvxs5K+nr7XMJB1r5H0gTLF9UKpL3JK+miaYXehpMclnZ3avyjp3WWIY9DvUdIhkpZLquvT/lj6/lB/631Y0jcHs08bGZxsLFfSFyd/BPy/iGgHZgKNwJdK9G0Ygv3t0zYktQF/DbwtIo4hm716IUBEfCEifrqvMe6r4vcYES+Qfffl7UXLjwSaIuKBykdnw4WTjeXNu4AtEfE9gIjoBv4c+Kik8ekv7Jsk/SfZRIOS9E1JT0r6CUWTUko6QdLP02SEdxRN8dIh6e8l/Rz4pKT9JN0s6cH0eGvqN1XSnZIelfRtSs+NNR3YCBSmsumMiOfT+jvOQNJZ0d9IeiSdBR2Z2vdTdk+aRyR9W9KLkqalM5Di+5h8RtIlfXcu6Qsp5sclXVk0y0Gv99hntevJvv1esGPyUUnvUXavpkcl/VRSS4l99jqzktRZ9PwvUzwLJf1Napsg6SfKJjF9XNLvlziOlnNONpY3RwMPFzdENtHoS8ARqekUYF5EvItsapE3AbOBjwG/DTvmj/sG8IGIOAG4mt5nR5Mi4r9FxNeArwOXRsRbgPcD3019LgZ+GRFvJvtW+UEl4v0NsBp4XtL3JL1nN+/t1cgmR7wC+EzRPu5O7T/uZx+7882IeEtEzALGkc1DVuo9FrsReG/RGc/vk02XD/BL4OT0nn8AfHaggUg6DWgnm4frOOAEZRNAngGsiIhjU5y3791btDzY52EGswoTpWfSLW6/KyIKE6C+A7g+nQGtkHR3an8TMAu4K/2xX0829UdB8SSG7waO0s5bE01Mc0u9g2zCQyLiJ5LW9Q0qIrqVzQH2FrJ74Vwq6YSIuKTEeyhMmvpwYbtk0w/9XtrW7aX2sQfvlPRZsul2ppBNT/KfJd5jccyrJD1Bdj+m1cD2iCicRbUBN6SzwNHA83sRy2np8Wh63UiWfH4BfFXSV8huQPeLvdim5YSTjeXNE2RnFztImkg22+6zwAnApj7r9JecnoiIU/rZT/E26oBTIuKNPvvtb9u9d57NCfUA8ICku4Dvkd35sa+t6Wc3O/9t9nfzvS56j0yM7dtB0ljgX8nmlFuWhtmK+/U9TsUKQ2mr6T356DeAf46I+ZLm9vM+dsSWhu0KxRsC/iEivl0i1hPI5hX7B0l3RsQXdxOb5ZCH0SxvfgaMl3QegKR6snuLXBOlZw6+FzhHUn36a/ydqf1pYD9Jp6TtjFL/Nzy7k2yiTVLf44q2fW5qO5Ps9sq9SDpQ0vFFTccBe3OXz1+S3XGxMAxV2MdqYHq6bjSG3sNjBYXE8qqy+xvtTYXazWQf/sVDaJDdNuPl9Hxe35WSF8iSPmR3qhyVnt9Bdm2tMb2fVknTJR0IbI6Ifwe+SnarYxtmfGZjuRIRIen3gH+V9H/J/mC6Ffirflb5MVlRwSKy2Xh/nrazLV3EvkxSM9m/hX8hO3Pq68+AyyUtTP3uJZvx+W+A6yU9krb7Uol1R5ENER0IbAFeSesOVGEfv5/2sRLYGBHbJX2R7C6szwNP9V0xIl6X9J303l8gmwJ/QNK6C8ju+lk8VHYJcJOkl8lmtz60xOrfAW6R9ADZHweb0jbvlPRbwH3prLAT+F9k19r+SVIP2QzGfzzQOC0/POuzWQ1LZy3dEdGVzsKuiIjj9rSeWa3xmY1ZbTsIuFHZlyy3kVXUmeWOz2zMzKzsXCBgZmZl52RjZmZl52RjZmZl52RjZmZl52RjZmZl9/8Bs4Ki+i2Y3mgAAAAASUVORK5CYII=
" style="border: 0px; box-sizing: border-box; height: auto; max-width: 100%; vertical-align: middle;" /></div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: row; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="text_cell_render border-box-sizing rendered_html" style="border-style: none; box-sizing: border-box; outline: none; overflow: auto hidden; padding: 0.5em 0.5em 0.5em 0.4em; resize: none; width: inherit;">
<div style="box-sizing: border-box;">
This is the all important step, now we create the synthetic control model.</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: column; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="input" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
In [9]:</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="input_area" style="background-color: #f7f7f7; background-position: initial initial; background-repeat: initial initial; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython3" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow-wrap: break-word; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all;"><span style="box-sizing: border-box;"></span><span class="n" style="box-sizing: border-box;">singvals</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="mi" style="box-sizing: border-box; color: #666666;">4</span>
<span class="n" style="box-sizing: border-box;">rscModel</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">RobustSyntheticControl</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">caStateKey</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">singvals</span><span class="p" style="box-sizing: border-box;">,</span> <span class="nb" style="box-sizing: border-box; color: green;">len</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">trainDF</span><span class="p" style="box-sizing: border-box;">),</span> <span class="n" style="box-sizing: border-box;">probObservation</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="mf" style="box-sizing: border-box; color: #666666;">1.0</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">modelType</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'svd'</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">svdMethod</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'numpy'</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">otherSeriesKeysArray</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="n" style="box-sizing: border-box;">otherStates</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">rscModel</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">fit</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">trainDF</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">denoisedDF</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">rscModel</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">model</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">denoisedDF</span><span class="p" style="box-sizing: border-box;">()</span>
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: row; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="text_cell_render border-box-sizing rendered_html" style="border-style: none; box-sizing: border-box; outline: none; overflow: auto hidden; padding: 0.5em 0.5em 0.5em 0.4em; resize: none; width: inherit;">
<div style="box-sizing: border-box;">
Next up, getting the counterfactual predictions and model fit.</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: column; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="input" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
In [10]:</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="input_area" style="background-color: #f7f7f7; background-position: initial initial; background-repeat: initial initial; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython3" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow-wrap: break-word; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all;"><span style="box-sizing: border-box;"></span><span class="n" style="box-sizing: border-box;">predictions</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="p" style="box-sizing: border-box;">[]</span>
<span class="n" style="box-sizing: border-box;">predictions</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">np</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">dot</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">testDF</span><span class="p" style="box-sizing: border-box;">[</span><span class="n" style="box-sizing: border-box;">otherStates</span><span class="p" style="box-sizing: border-box;">],</span> <span class="n" style="box-sizing: border-box;">rscModel</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">model</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">weights</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">actual</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">dfProp99</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">loc</span><span class="p" style="box-sizing: border-box;">[</span><span class="n" style="box-sizing: border-box;">dfProp99</span><span class="p" style="box-sizing: border-box;">[</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'LocationDesc'</span><span class="p" style="box-sizing: border-box;">]</span> <span class="o" style="box-sizing: border-box; color: #666666;">==</span> <span class="n" style="box-sizing: border-box;">caStateKey</span><span class="p" style="box-sizing: border-box;">]</span>
<span class="n" style="box-sizing: border-box;">actual</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">actual</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">drop</span><span class="p" style="box-sizing: border-box;">(</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'LocationDesc'</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">axis</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="mi" style="box-sizing: border-box; color: #666666;">1</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">actual</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">actual</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">iloc</span><span class="p" style="box-sizing: border-box;">[</span><span class="mi" style="box-sizing: border-box; color: #666666;">0</span><span class="p" style="box-sizing: border-box;">]</span>
<span class="n" style="box-sizing: border-box;">model_fit</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">np</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">dot</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">trainDF</span><span class="p" style="box-sizing: border-box;">[</span><span class="n" style="box-sizing: border-box;">otherStates</span><span class="p" style="box-sizing: border-box;">][:],</span> <span class="n" style="box-sizing: border-box;">rscModel</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">model</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">weights</span><span class="p" style="box-sizing: border-box;">)</span>
</pre>
</div>
</div>
</div>
</div>
</div>
<div class="cell border-box-sizing text_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: row; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="text_cell_render border-box-sizing rendered_html" style="border-style: none; box-sizing: border-box; outline: none; overflow: auto hidden; padding: 0.5em 0.5em 0.5em 0.4em; resize: none; width: inherit;">
<div style="box-sizing: border-box;">
And finally, the plot! That's it. You can clearly see the impact that Prop99 had, via the sharper decline in per capita smoking compared to the counterfactual.</div>
</div>
</div>
</div>
<div class="cell border-box-sizing code_cell rendered" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; background-image: linear-gradient(to right, transparent -40px, transparent 1px, transparent 1px, transparent 100%); border-radius: 2px; border: 1px solid transparent; box-sizing: border-box; display: flex; flex-direction: column; margin: 0px; outline: none; padding: 5px; text-size-adjust: auto; width: 1110px;">
<div class="input" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row;">
<div class="prompt input_prompt" style="-webkit-user-select: none; border-top-color: transparent; border-top-style: solid; border-top-width: 1px; box-sizing: border-box; color: #303f9f; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
In [11]:</div>
<div class="inner_cell" style="-webkit-box-align: stretch; -webkit-box-flex: 1; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; flex: 1 1 0%; min-width: 0px;">
<div class="input_area" style="background-color: #f7f7f7; background-position: initial initial; background-repeat: initial initial; border-bottom-left-radius: 2px; border-bottom-right-radius: 2px; border-top-left-radius: 2px; border-top-right-radius: 2px; border: 1px solid rgb(207, 207, 207); box-sizing: border-box; line-height: 1.21429em;">
<div class=" highlight hl-ipython3" style="background-color: transparent; background-position: initial initial; background-repeat: initial initial; border: none; box-sizing: border-box; margin: 0.4em; padding: 0px;">
<pre style="background-color: transparent; border-radius: 2px; border: none; box-sizing: border-box; color: #333333; font-size: inherit; line-height: inherit; overflow-wrap: break-word; overflow: auto; padding: 0px; white-space: pre-wrap; word-break: break-all;"><span style="box-sizing: border-box;"></span><span class="n" style="box-sizing: border-box;">fig</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">ax</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">subplots</span><span class="p" style="box-sizing: border-box;">(</span><span class="mi" style="box-sizing: border-box; color: #666666;">1</span><span class="p" style="box-sizing: border-box;">,</span><span class="mi" style="box-sizing: border-box; color: #666666;">1</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">tick_spacing</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="mi" style="box-sizing: border-box; color: #666666;">5</span>
<span class="c1" style="box-sizing: border-box; color: #408080; font-style: italic;"># this is a bug in matplotlib</span>
<span class="n" style="box-sizing: border-box;">label_markings</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">np</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">insert</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">years</span><span class="p" style="box-sizing: border-box;">[::</span><span class="n" style="box-sizing: border-box;">tick_spacing</span><span class="p" style="box-sizing: border-box;">],</span> <span class="mi" style="box-sizing: border-box; color: #666666;">0</span><span class="p" style="box-sizing: border-box;">,</span> <span class="s1" style="box-sizing: border-box; color: #ba2121;">'dummy'</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">ax</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">set_xticks</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">np</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">arange</span><span class="p" style="box-sizing: border-box;">(</span><span class="nb" style="box-sizing: border-box; color: green;">len</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">label_markings</span><span class="p" style="box-sizing: border-box;">)))</span>
<span class="n" style="box-sizing: border-box;">ax</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">set_xticklabels</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">label_markings</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">rotation</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="mi" style="box-sizing: border-box; color: #666666;">45</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">ax</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">xaxis</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">set_major_locator</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">ticker</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">MultipleLocator</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">tick_spacing</span><span class="p" style="box-sizing: border-box;">))</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">plot</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">years</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">actual</span> <span class="p" style="box-sizing: border-box;">,</span><span class="n" style="box-sizing: border-box;">label</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'actual'</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">xlabel</span><span class="p" style="box-sizing: border-box;">(</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'Year'</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">ylabel</span><span class="p" style="box-sizing: border-box;">(</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'Per capita cigarette consumption'</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">plot</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">trainingYears</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">model_fit</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">label</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'fitted model'</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">plot</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">testYears</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">predictions</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">label</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'counterfactual'</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">title</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">caStateKey</span><span class="o" style="box-sizing: border-box; color: #666666;">+</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">', Singular Values used: '</span><span class="o" style="box-sizing: border-box; color: #666666;">+</span><span class="nb" style="box-sizing: border-box; color: green;">str</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">singvals</span><span class="p" style="box-sizing: border-box;">))</span>
<span class="n" style="box-sizing: border-box;">xposition</span> <span class="o" style="box-sizing: border-box; color: #666666;">=</span> <span class="n" style="box-sizing: border-box;">pd</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">to_datetime</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">yearTrainEnd</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">errors</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'coerce'</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">axvline</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">x</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="nb" style="box-sizing: border-box; color: green;">str</span><span class="p" style="box-sizing: border-box;">(</span><span class="n" style="box-sizing: border-box;">yearTrainEnd</span><span class="p" style="box-sizing: border-box;">),</span> <span class="n" style="box-sizing: border-box;">color</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'k'</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">linestyle</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="s1" style="box-sizing: border-box; color: #ba2121;">'--'</span><span class="p" style="box-sizing: border-box;">,</span> <span class="n" style="box-sizing: border-box;">linewidth</span><span class="o" style="box-sizing: border-box; color: #666666;">=</span><span class="mi" style="box-sizing: border-box; color: #666666;">4</span><span class="p" style="box-sizing: border-box;">)</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">grid</span><span class="p" style="box-sizing: border-box;">()</span>
<span class="n" style="box-sizing: border-box;">plt</span><span class="o" style="box-sizing: border-box; color: #666666;">.</span><span class="n" style="box-sizing: border-box;">legend</span><span class="p" style="box-sizing: border-box;">()</span>
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column; position: relative; z-index: 1;">
<div class="output" style="-webkit-box-align: stretch; -webkit-box-orient: vertical; align-items: stretch; box-sizing: border-box; display: flex; flex-direction: column;">
<div class="output_area" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row; padding: 0px;">
<div class="prompt output_prompt" style="-webkit-user-select: none; box-sizing: border-box; color: #d84315; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0.4em; text-align: right;">
Out[11]:</div>
<div class="output_text output_subarea output_execute_result" style="-webkit-box-flex: 1; box-sizing: border-box; flex: 1 1 0%; line-height: 1.21429em; max-width: calc(100% - 14ex); overflow-x: auto; padding: 0.4em;">
<pre style="background-color: transparent; border-radius: 0px; border: 0px; box-sizing: border-box; font-size: inherit; line-height: inherit; overflow-wrap: break-word; overflow: auto; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-break: break-all;"><matplotlib.legend.Legend at 0x7f9478d14ac8></pre>
</div>
</div>
<div class="output_area" style="-webkit-box-align: stretch; -webkit-box-orient: horizontal; align-items: stretch; box-sizing: border-box; break-inside: avoid; display: flex; flex-direction: row; padding: 0px;">
<div class="prompt" style="-webkit-user-select: none; box-sizing: border-box; cursor: default; font-family: monospace; line-height: 1.21429em; margin: 0px; min-width: 14ex; padding: 0px 0.4em; text-align: right;">
</div>
<div class="output_png output_subarea " style="-webkit-box-flex: 1; box-sizing: border-box; flex: 1 1 0%; max-width: calc(100% - 14ex); overflow-x: auto; padding: 0.4em;">
<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAYgAAAElCAYAAAD+wXUWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4FFXfxvHvL500SEgInYTeE5IAAWkhKEW6KE0QUZoF24MgvmIvqA+PoCKCUkWKKL0joShFCL1JD4QaegKEtPP+MZsYMGUTstmU87muubI7O3Pm3k2yZ+bMzDmilELTNE3THmRj7QCapmla/qQrCE3TNC1duoLQNE3T0qUrCE3TNC1duoLQNE3T0qUrCE3TNC1duoIo4kREiUhV0+NJIvJOmteGicglEYkVkZK5vN2KpnJtc7Nca23HtK3TItLG0tsxh4gMEJE/rJ0jt4mIr+lv1s7aWYoCXUEUAiLSR0R2mr4IL4jIShFplt1ylFJDlVIfmsq0B8YBjymlXJVSV3Mzs1LqjKncpIctS0TKi8ivInJFRG6KyH4RGZDb28krIuIkIjdEpHU6r/1PRBZYI1dBJiLVRCRORH6ydpaCRFcQBZyIvA58BXwC+AAVgYlAl4cs2gdwAg7mIJOISF7+bc0CzgKVgJJAf+BSHm7/oTy4N6yUigPmYbyPtMvZAr2BGXmXrtD4Fthh7RAFja4gCjARKQ58ALyolPpNKXVbKZWglFqqlBphWqaRiGw17ZFeEJFvRMQhg/Kmi8hHIlId+Ns0+4aIrDe93lREdpj20neISNM0624QkY9F5E/gDlDZNO9DEflTRGJEZI2IeJmWv6+pQESeFZHDpuVOisiQbHwUDYHppvefqJTarZRamcF2Msxker2/iESKyFUReSdts1HK55Nm2VYiEpXBZ5np527K9KKIHAOOpVPEDOAJEXFOM68txv9synsbJSInTO/jkIh0yyDLv5plTJ/D82meDzR9/tdFZLWIVDLNF9NRy2XT732fiNTNYDv3NbGJyHspe+ymo6KfTJ/rDdPfj4/pteIi8qPpczpn+hu0Nb1mKyJfmo4OTwKPp7ftzIhIL+AG8Ht21y3qdAVRsDXB2MtfmMkyScBrgJdp+TDghcwKVUodBeqYnpZQSrUWEU9gOTABYy99HLBc7j830Q8YDLgBkaZ5fYBngVKAA/CfDDZ7GegIuJuW/5+IBGaWM41twLci0ktEKpqxfLqZRKQ2xtFXX6AMUBwoZ2aGB5nzuXcFGgO1H1xZKbUFuAB0TzO7H/CzUirR9PwE0NyU833gJxEpk92gItIVGG3aljewGZhjevkxoAVQHSgB9ARy0tz4jClnBYy/n6HAXdNrM4BEoCrQwLTNlMprEMbfRQMgGOjxQPZRIrIsk/fmjrET9UYOMhd5uoIo2EoCV9J8YfyLUipCKbXNtGd9GvgeaJmDbT0OHFNKzTKVNQc4AnRKs8x0pdRB0+sJpnnTlFJHlVJ3gflAQAY5lyulTijDRmANxpefOZ7E+FJ7BzglIntEpGEmy2eUqQewVCn1h1IqHhgD5KizMjM/90+VUtdMOdIzE1Mzk+mLrgtpmpeUUr8opc4rpZKVUvMwjkQa5SDuEFOWw6a/pU+AANNRRAJGhV8TENMyF3KwjQSMv9eqSqkk0+dzy3QU0R541XQEeBn4H9DLtN5TwFdKqbNKqWvAp2kLVUp9ppTqmMl2PwR+VEqdzUHmIk9XEAXbVcDrwTbstESkuogsE5GLInIL45/fK6PlM1GWf44KUkRy/x52ev+EF9M8vgO4ZpCzvYhsE5FrInID6GBuTqXUdaXUKKVUHYxzJ3uARSIiGaySUaayad+DUuoOOdtbNvdzz+pLayYQKiLlMCqv40qp3Wm20d9UGd4wfWZ109mGOSoB49OUcw0QoJxSaj3wDUYb/iURmWyqrLJrFrAamCsi50XkczEuhKgE2AMX0mz/e4yjO3jgd8K//wYzJCIBQBuMCkfLAV1BFGxbgTiMpoqMfIexp19NKeWO0ZSQ0RdnZs5j/DOnVRE4l+Z5jva2RcQR+BX4EvBRSpUAVuQkp1LqiqmcsoBnNle/AJRPk6sYxl5vittA2nMCpTMpy5zPPdPPSyl1BuPIqC9G89LMNNkqAVOAl4CSps/sQDrbSMlNJtnPAkOUUiXSTMVMzVwopSYopYIwmh2rAyMyiJzh52M6N/a+Uqo20BSj2ai/adv3AK8023Y3VfZg/E4qpCnTnCbEFK0AX+CMiFzEaEp8QkR2ZaOMIk1XEAWYUuomRjPItyLSVUScRcTetDf+uWkxN+AWECsiNYFhOdzcCqC6GJfU2olIT4y28wzbf7PBAXAEooFEEWmP0Q6dynSStVV6K4vIWBGpa8rlhvEej+fg0twFQCcxTsY7YLTrp/3C3QN0EBFPESkNvJpJWbn1uc/AqAQeAWanme+CUcFEg3GSH+MI4l+UUtEYFfnTppO+A4EqaRaZBLwlInVMZRUXkSdNjxuKSGPT3v5tjB2SjC4Z3gP0Mv0N3ne+QERCRaSe6eTzLYwmpyRTc9Ua4L8i4i4iNiJSRURSmuPmA8PFuJTZAxiV9UeWarLpfQaYpkkY59HaZqOMIk1XEAWcUmoc8DrwfxhfFmcxvlAWmRb5D8ZJ2RiMPc55OdzOVYy9vjcwml3eBDqa9tgfilIqBhiO8WVw3ZR3ScrrIlIeiAX2Z1CEM8aJ+hvASYwjnc45yHEQeBmYi7HnGoNx8vyeaZFZwF7gNMaXWmafZa587hiVlgfwe9q2f6XUIeC/GEeRl4B6wJ+ZlDMIY8//KsaRwJY0ZS0ExmI0/9zCOBJpb3rZ3ZT/OkbzzlWMI7T0vIPxhXwdo3L9Oc1rpU3v5RZwGNgIpNyT0B9jJ+GQad0FGBcJYNr2aozPfRfwW9oNishoEVmZXhil1B2l1MWUCeNvKM5UYWpmED1gkJbficjTQB2l1Ft5vF1XjEqnmlLqVF5uW9PyA11BaFoaItIJ43p5wdhDbwwEKv2PohVBuolJ0+7XBeOE/HmgGtBLVw5aUaWPIDRN07R06SMITdM0LV0FustcLy8v5evrm6N1b9++jYuLS+4GyiGdxXo5IiIi7nseFBRktSxZyS85QGfJzzkg6ywRERFXlFLeWRaklCqwU1BQkMqp8PDwHK+b23SWf8urHBj3EqRO1sySlfySQymdJT35JYdSWWcBdiozvmN1E5OmaZqWLl1BaJqmaenSFYSmaZqWrgJ9klrTtLyVkJBAVFQUxYsX5/Dhw9aOA5BvsuSXHPBPFicnJ8qXL4+9vX2OytEVhFakKX0fULZERUXh5uZGyZIlcXfPSa/fuS8mJgY3Nzdrx8g3OcDI4urqytWrV4mKisLPzy9H5egmJk3TzBYXF0fJkiXJeKgNLb8QEUqWLElcXFyOy9AVhKZp2aIrh4LjYX9XuoLIr5SCiOlwdI21k2iaVkTpCiI/UgpWj4alr8Dc3nBsrbUTaVqBtGHDBrZs2ZL1gplwdU13lNwiQVcQ+U1yEix5GbZNhIbPQ6laML8/RO20djJNK3Byo4IoynQFkZ8kxsOvz8HuWdDiTejwJfT9FVxLwewnIfqotRMWOoMHD75v0gqGrl27EhQURJ06dZg2bRoAq1atIjAwEH9/f8LCwjh9+jSTJk3if//7HwEBAWzevJkBAwawYMGC1HJSjg5iY2MJCwsjMDCQevXqsXjxYqu8r/xGX+aaX8TfQc3vjxxfy84arzPjUgciPluPt7sTLwR+x2Pb+iOzusFza6B4OWunLTSmTJly3/PJkydbKUnB8/7Sgxw6fytXy6xd1p13O9XJcrmpU6fi6enJ3bt3CQoKomfPngwaNIhNmzbh5+fHtWvX8PT0ZOjQobi6uvKf//wHgB9//DHd8pycnFi4cCHu7u5cuXKFkJAQOnfuXORPyOsjiOy4FwORWyE5+aGLir2XyPHLsfxx7ArrT9zk73FtUcfXMSrheXrsDeavU1dpUNGD2LgEhiy/ztNxI7h3+zoJM7vBnWu58GY0reCaMGEC/v7+hISEcO7cOSZPnkyLFi1Sr/f39PTMVnlKKUaPHk39+vVp06YN586d49KlS5aIXqDoIwhzJSfDvKfh5AYoWRUaDwX/3uCY+QkspRQr9l9k/ZHLXLx1l4s347h06x6x9xIBKEEMMxzGUsUmkjnlxxDc4Cle8PWkgmcxRASlFH8cv8KMLaV49u9XmZ44lhMTOnGjx3wCq5Qt8ns4mvWYs6dvCRs2bGDdunVs3boVZ2dnmjdvjr+/P3///XeW69rZ2ZFs2sFTShEfHw/A7NmziY6OJiIiAnt7e3x9fR/q/oHCQlcQ5toy3qgcGg6CcxGw4j+w/kMIfAYaDYYSFf61ysnoWMYsPsgfx6/g5epIeY9iVPdxo3lVL+rLcRpcX0mFcyuQhLvY9vqZvjXa/asMEaF5NW+aV/PmzNU6LF3hTLcT/8f6GX2pnfwfirsUw8PFAU8XezxdHPF0tsfL1ZHuQeUpV6JYHnwwmpa3bt68iYeHB87Ozhw5coQdO3Zw7949Nm7cyKlTp+5rYnJzc+PWrX+awXx9fYmIiOCpp55i8eLFJCQkpJZZqlQp7O3tCQ8PJzIy0lpvL1/RFYQ5onbC+o+gdlfo8IVp3g7jSqOt3xpTrU4Q0Ac8/IhzLs3EPy8waeNJHO1t+LBLHfo0roRt7AXYOxf2zoErR8GuGNTqxE6HxgSnUzk8qGJJZyr2e4n4rQ60WT2CZZ5fM6f0CE7HO3Htdjz7r9/g2u14bsUlMj/iLL8Oa0opNycLfzialrfatWvHpEmTqF+/PjVq1KBhw4Z4e3szefJkunfvTnJyMqVKlWLt2rV06tSJHj16sHjxYr7++msGDRpEly5daNSoEWFhYamD6vTt25dOnToRHBxMQEAANWvWtPK7zB90BZGVuJuw4FlwKwudxkNKk06FRsZ04yzsmGLc1HZoEQBOwEDlQk/X0niVq4zj1Yow+zScDAeVDBWbQOevjQrHyZ3YDRuyFcmhyWCws6HK6v/j/04PhPafGc1dpmx7zt6g9+RtDJy+g7mDm+DqqH/NWuHh6OjIypUrU5+n7QOpffv29y1bvXp19u3bd9+8bdu2pT7+9NNPAfDy8mLr1q3pbi82NjZXchdE+psjM0rB0lfh5jkYuAqKlfj3MiUqkNj6PY5UHcbqdSs5c+oYdV1j6OSbTDmuGute2AmObtD8DeOLvGSVh8/W8HmoHAqLXoBFw+DgIqMCcy9DQIUSTOwbyPMzdzLspwh+fKYhDnb6egRN07JHVxCZ2T0LDv4GYWOMowWTuIQk9kXd5K9TV/nr9HV2RV4n9l4ijnZlGP5oC/o398PRztby+UpWgWdXwPbv4fcPYGJjaDcW/HsRWrMUn3avx5sL9jHy133890l/bGz0CW1N08xXZCuILLt5jv4bVrwJfi3hkdcAWLH/AtP/PM2eszeITzKuhKjh40bXBmVp6OtJ0ypeeLs5Wjr6/WxsockLUL2t6WhiqNHU1WkCTwVX4NLNOP679iil3B15q32tvM2maVqBViQriMMXbvHe1jik7GVaVvf+96WiCXGwYCA4uED3yVy9k8CYxQdZvv8CVbxdeKZpJRr5lSS4kgceLg7WeRMPevBo4vsW0PMnXmodzMVbcXy/8SSl3Z149pGc9QuvaVrRUyQriOt34rmToBgwbQchlT15s11NAit6/LPAmv+DSweg7wKWn1K8s3gTMXEJjGhbgyEtKmNnm0/b81OOJvxawNw+ML0D8vg4PujyNNEx9/hg2SFKuTnxeP0y1k6qaVoBkE+/6SyraRUvPm1ejPc71+H45Vi6T9zC4Jk7OX45BnbPhh1TuBs0lBf+8uTFn3dR3qMYy15uzouhVfNv5ZBW6boweANUagpLXsJ25QgmPFWXoIoevDZvDz/+cYrbphv1NE3TMmKxbzsRmSoil0XkQJp5X4jIERHZJyILRaREmtfeEpHjIvK3iLS1VK4UdjbCM0192TgilNcfrc6WE1eZN2EkLH6BSyVDaLm7JesOXWZE2xr8NqwpNUrnj6EEzebsaXT01+Ql2DEFpznd+bFHJQIrleDDZYdo8unvfL7qCJdv6btFtYJlwoQJ1KpVi759+7JkyRLGjRsHwKJFizh06FDqctOnT+f8+fPZKvv06dPUrVs3V/Omx5wuxPNDN+OW3B2eDjx499daoK5Sqj5wFHgLQERqA72AOqZ1JopIHlwGBC6OdgxvXZUdIX/wtt1sViY3pvm5YZT2dGfZ8GYF56ghPbZ20PZj6P4DnIug+KzHmNvRid9eaEqzal5M2niCR8auZ8Qvezl6KcbaaTXNLBMnTmTFihXMnj2bzp078/rrrwO5U0Fo97PYOQil1CYR8X1gXtrh0bYBPUyPuwBzlVL3gFMichxoBKR/50puSkqEZa9SbPcsCHqW+o98xHeXbtOyunfBrRgeVP9J8Kpm9CU1tR2BjQYx8bH+RLarydQ/TjF/ZxS/RETRqoY3Qa6JtEhWReaS2J079TgbBcnQoUM5efIknTt3ZuDAgXh4eLBlyxYGDBjAkiVL2LhxIx999BG9e/dm586d9O3bl2LFirF161YOHTrE66+/TmxsLF5eXkyfPp0yZcoQERHBwIEDcXZ2plmzZulud8OGDbz77rv4+PiwZ88eunfvTr169Rg/fjx3795l0aJFlCpVisjISAYOHEh0dDTe3t5MmzaNihUrcurUKfr06UNiYiLt2t2/3/zFF18wf/587t27R7du3Xj//ffz4qM0izVPUg8E5pkel8OoMFJEmeZZVkKcMf7CkWXG+AuhoyknQjlP6x/a5bqyAcZ5iRUjYNt3sOVrKlVswvuB/Xm1ZQdm74pm+pZINsTeY2HkRp5p6ssTQeX/uQs7KdE4IilkgoKCrB2h4Fo5Ci7uz90yS9czegbIwKRJk1i1ahXh4eGpX/IATZs2pXPnznTs2JEePYz9zpUrV/Lll18SHBxMQkICL7/8MosXL8bb25t58+bx9ttvM3XqVJ599lm+/vprWrZsyYgRIzLc9t69ezl8+DCenp5UrlyZ559/nr/++ovx48fz9ddf8+GHH/LSSy/Rv39/nnnmGaZOncrw4cNZtGgRr7zyCsOGDaN///58++23qWWuWbOGY8eO8ddff6GUonPnzmzatIkWLVrkzuf5kLL8jxeR7sBYoBQgpkkppdxzulEReRtIBGanzEpnsXRvVBCRwcBgAB8fHzZks5uKFHE3LnPjmzBK3DzAsaqDOGfzCGzcmKOyHlZsbGyO30e2eT+DffEulL64njIX1uK8aBhuts5082lBozqtOHAhluhrV7i54iIbVl2mtsNlynEJh8TbXCzdmlN+TxPv6JH1dh5Snn4mWcgvWfJDjuLFixMTE0NSUhLxCfHYJOXuxQ7JCfHci8m8uVMpRWxsLI6OjsTFxaGUIiYmhoSEBO7evUuMaf2kpCRu375NTEwMhw4d4sCBA4SFhaW+5uPjQ1RUFNevXycwMJCYmBi6d+/O8uXLU8tIcefOHQIDA3F1dSU+Ph5fX1+aNWtGTEwMVapUYe3atSQlJbFlyxZmzJhBTEwMXbt2ZcSIEcTExPDHH38wffr01PkjR44kJiaGZcuWsXr1avz9/QHjd7x//34aNGgA8K8c5kpKSkpdNy4uLsd/N+bsEn4OdFJKHc7RFh4gIs8AHYEw9c/dalFA2u5QywPpNh4qpSYDkwGCg4NVq1atsh8i9jIxk17F7c5ZeOJHqtXrQbXsl5JrNmzYQI7ex0PpanQlcmYrdrtmUu7gQsqdX0XK/eLK3oZrdj4cuVeSrcmNKFfckZaX11Lm6lZo/ppx8tvecr3FWuczSV9+yZIfchw+fBg3NzdiYmJw6DzOItvI6s4iEcHV1RU3NzecnJwQEdzc3LC3t6dYsWKp/TLZ2tri4uKCm5sbzs7O1KlT51/9Ld24cQMbG5vUdVxcXO57nsLZ2RlnZ+fU+fb29nh4eODm5pZ6MtnW1va+LAkJCalliQju7u7Y2dml3qSbstzo0aMZMmRIuu/1wRzmSts/lZOTU2qFk13mNLJfysXKoR0wEuislLqT5qUlQC8RcRQRP6Aa8FdubDNdkVsodvcC9J4H9XpkvXxhJWJcCtttErzxN/SYyr5678LLu5C3L1Hy7SNUH7Geq6FjGX57AN0Yx/UyzYyebb8Ohn3zc2XwJE3LDSkVV3rPa9SoQXR0dGoFkZCQwMGDBylRogTFixfnjz/+AIxxIR5G06ZNmTt3bmpZKec0Hnnkkfvmp2jbti1Tp05N7RDw3LlzXL58+aEy5CZzKoidIjJPRHqLSPeUKauVRGQOxknmGiISJSLPAd8AbsBaEdkjIpMAlFIHgfnAIWAV8KJSKimnbypLdbqyvfH3UK2NxTZR4BQrAXWf4FrJQOOubDtjP87bzZHhYdVY+OIjXC9WgcYnn2VL8xngUhJ+GwQ/hMGZbVkUrmmW16tXL7744gsaNGjAiRMnGDBgAEOHDiUgIICkpCQWLFjAyJEj8ff3JyAggC1btgAwbdo0XnzxRZo0aUKxYg93VDxhwgSmTZtG/fr1mTVrFuPHjwdg/PjxfPvttzRs2JCbN2+mLv/YY4/Rp08fmjRpQr169ejRo0eOm5UsQbLqk0hEpqUzWymlBlomkvmCg4NVTq9CyQ+H6ykKSpZrt+MZMmsnO05f5402VXnJKwL5/QOIuQA9Z0OtjnmSIzc9OAb14MGDrZYlK/khx+HDh6lVq9Z9TRjWll+y5JcccH+WlN9ZWiISoZQKzqqcLM9BKKWezWlIrXDxdHHgp+cbM+rX/fx33XFOBdbm02F/4fhTF6PLce+a4FXV2jGz5cG23/QqCE0rqrJsYhKR8qa7ni+LyCUR+VVEyudFOC3/cbSzZdxT/rzWpjq/7TpHv1kHudFpKtjaw7y+cK/oDq6iaYWNOecgpmGcRC6LcW/CUtM8rYgSEV5pU43xvQLYc+YG3WZHcr3D98YwqkteMq6O0jStwDOngvBWSk1TSiWapumAt4VzaQVAl4ByzB7UmIs34xi4yZmE0DFwcKExRremaQWeORXEFRF5WkRsTdPTwFVLB9MKhoa+nvz3KX92n7nBWxdDUbU6wdoxcGqztaNpmvaQzKkgBgJPAReBCxj9J1n9CiYt/+hQrwzDW1dlwa5zzPIZaVwmu+BZYzxuTdMKrCwrCKXUGaVUZ6WUt1KqlFKqq1IqMi/CaQXHq22q81htH95bfYadIRMg4S788gwk3rN2NE3Llq+++oo7d+5kveADjh49SkBAQOp9GNmxYcOG1PsycsoS3YNnWEGIyJumn1+LyIQHp1xPohVoNjbCuJ4BVC3lynPLY7jcehxE7YBVb1k7mqZlS04qiKSkJJYtW0aXLl3YvXs3VapUydb6uVFBWEJmRxAp3WvsBCLSmTTtPq6OdvzQvyEi0GdLaeIbvwQ7f4QT4daOphUyM2fOpH79+vj7+zNo0CAiIyMJCwujfv36hIWFcebMGQAGDBjAggULUtdL2ctOuemwR48e1KxZk759+6KUYsKECZw/f57Q0FBCQ0MBo8fVJk2aEBgYyJNPPpnaLYavry8ffPABzZo1Y968eUycOJEffvghdb2uXbsSFBREnTp17rshc9WqVQQGBuLv709YWBinT59m0qRJ/O9//yMgIIDNmzdnmDs2NpawsDACAwOpV68eixcvtuCnnMmNckqppaaHd5RSv6R9TUSetGgqrcCqWNKZiX0C6Tf1L4Zf7MB3HsuQVW/B0D8KZXfhRdnYv8Zy5NqRXC2zpmdNRjYamekyBw8e5OOPP+bPP//Ey8uLyMjIDLvZzszu3bs5ePAgZcuW5ZFHHuHPP/9k+PDhjBs3LrU78StXrvDRRx+xbt06XFxcGDt2LOPGjWPMmDGA0RFeSj9O+/fvp2TJkvznP/8BYOrUqXh6enL37l0aNmzIE088QXJyMoMGDWLTpk34+flx7do1PD09GTp0KK6urqnr/vjjj+lmdnJyYuHChbi7u3PlyhVCQkLo3LkzIpYZv8Wck9TptRHodgMtQ02revHO47VY9fcNFpV6AaIPQ4S+dUbLHevXr6dHjx54eXkB4OnpydatW+nTpw8A/fr1S/3SzkyjRo0oX748NjY2BAQEcPr06X8ts23bNg4dOsQjjzxCQEAAM2bMIDLyn1OwPXv2zLD8CRMm4O/vT0hICGfPnuXYsWNs27aNFi1a4Ofnl5o9O5RSjB49mvr169OmTRvOnTvHpUuXslVGdmS4Syci7YEOQLkHzjm4Y4zloGkZeqapL0cuxvDaDkWob1NKhH8MdZ8wxsrWCoWs9vQtRSmV5R5zyut2dnYkm3ocVkoRHx+fuoyjo2PqY1tbWxIT//21ppTi0UcfZc6cOelux8XFJd35GzZsYN26dWzduhVnZ2datWqVOnaFOXv7GeWePXs20dHRREREYG9vj6+vL3FxlhtXPrMjiPMY5x/iuP/cwxKgrcUSaYWCiPBupzpU8HTmjVs9UXE3YUPGI4VpmrnCwsKYP38+V68at2Ndu3Ytw262fX19iYgwTpkuXryYhISELMtP2014SEgIf/75J8ePHweMgYOOHj2aZRk3b97Ew8MDZ2dnjhw5wrZtRo/HTZo0YePGjZw6dSo1+4PbzCz3zZs3KVWqFPb29oSHh993NGMJGVYQSqm9SqkZQFVgDrAb2AUsU0pdt2gqrVAo5mDLR13r8fs1b/aVfgJ2/ACXc2VoEa0Iq1OnDm+//TYtW7bE39+f0aNHZ9jN9qBBg9i4cSONGjVi+/btGe7xpzV48GDat29PaGgo3t7eTJ8+nd69e1O/fn1CQkI4ciTr8y7t2rUjMTGR+vXr88477xASEgKAt7c3kydPpnv37vj7+6c2UXXq1ImFCxemnqTOKHffvn3ZuXMnwcHBzJ49m5o1a+b0YzSLOd19dwC+B05gDA3qBwxRSq20aDIz6O6+c58lsgyfs5ttB46x1XUEtuUaQL+FxmBFeZwjPbq77+zR3X3n/xyQh919A+OAUKXUcVPBVYDlgNUrCK1g+L+OtWjz92VmOfZhwMmJ8Pe0d2ZbAAAgAElEQVRKqNnB2rEA3b23pmXGnKuYLqdUDiYngfwzJp6W75Vyc2JU+1p8dDmEm65VYPVofYe1phUA5lQQB0VkhYgMEJFnMLr73mHu0KOaBtCrYQX8K3kz8nZvuH4Ktn1n7UhaDmXVLK3lHw/7uzKngnACLgEtgVZANOAJdAJyb4xJrVCzsRE+6VaPdffqcMDtEdj0BcRY7vptzTKcnJy4evWqriQKAKUUV69excnJKcdl6CFHtTxTo7Qbg1tU5qWN3Vlf7C9s1r0H3fSRREFSvnx5oqKiuHHjxkN98eSmuLi4fJElv+SAf7I4OTlRvnzOBwDNsoIQET/gZcA37fJKqc453qpWZL3cuhrL9l1gXkIneu/9GWxsoMOXYF/MKnlSrjVPERQUZJUcBYW9vT1+fn5s2LCBBg0aWDsOQL7Jkl9yQO5lMecqpkXAjxjnHpIfeotakWbcG1GXAVO741e1BCG7p8L5vfDUDGMciTwWHHz/lX666UTT/mHOOYg4pdQEpVS4UmpjymTxZFqh1aK6N50CytPreBsmlv2UpBtnYXIrOLw0y3U1Tcs75lQQ40XkXRFpIiKBKZPFk2mF2qfd6zE8rBrfRFWmVcwHnLUpB/OehtVvQ1LW3SFommZ55jQx1QP6Aa35p4lJmZ5rWo44O9jx+qPV6RdSiW/Dj9Nu+1uMtv2Jvlu/IfHMDhwqDrF2RE0r8sypILoBlZVS8VkuqWnZ5O3myHud6zDwET/Gra3IX/ur8em5H6h34TVUrXJIxRBrR9S0IsucJqa9QAlLB9GKtoolnfmqVwOGvDiKD8t8y9VEJxKndSR+50/WjqZpRZY5FYQPcEREVovIkpTJ0sG0oql2WXc+GdyDaRU/ZXtSdRyWvUjskpGQnGTtaJpW5JjTxPRuTgoWkakYd1pfVkrVNc3zBOZh3FNxGnhKKXVdjBE0xmMMUHQHGKCU2pWT7WoFn4gQWrUk8U3m8/P81+mzaxI3L/9N8adngFNxa8fTtCIjyyOItJe2ZvMy1+lAuwfmjQJ+V0pVA343PQdoD1QzTYMBfXutRus65Ql+4Ue+tB+G89lN3PqmFVw9Ye1YmlZkZFlBiEiMiNwyTXEikiQit7JaTym1Cbj2wOwuwAzT4xlA1zTzZyrDNqCEiJQx/21ohVV1Hzeee/UDPvX6hKSYy9yd2IqkExusHUvTioQsBwz61woiXYFGSqnRZizrizECXUoT0w2lVIk0r19XSnmIyDLgM6XUH6b5vwMjlVL/Gg1IRAZjHGXg4+MTlDLMYHbFxsbi6uqao3Vzm86SdY7EZMXaA2cZeGUslW0usrXamySXa/zQ2wkNDb3veXh4eJZZrCW/5ACdJT/ngKyzhIaGmjVgEEqpbE/ANjOX8wUOpHl+44HXr5t+LgeapZn/OxCUVflBQUEqp8LDw3O8bm7TWf4toxzzN+9T+8c0UPfGeKhff/pOxcQlPNR2MO7pSZ2ykyWv5ZccSuks6ckvOZTKOguwU5nxHW5OZ31px3ywAYJN/0w5cUlEyiilLpiakFIGHooCKqRZrjxwPofb0AqxJ5vVI7rGSi5M7Uqno6N55/NoGnd8lq4B5ZAshjHVNC17zLnMtVOaqS0Qg3HOICeWAM+YHj8DLE4zv78YQoCbSqkLOdyGVsh5e/tQ6ZXV3PNpwMdJ41j/y3f0mLSV/VE3rR1N0woVi40HISJzMAYY8hKRKIzLZT8D5ovIc8AZ4EnT4iswLnE9jnGZqx6DQsuckzuuzy1GzX6S8Wcm8m60DZ2/vc5rbaozPKya2cUMGjTIgiE1rWAzp4npc+Aj4C6wCvAHXlVKZXqLq1KqdwYvhaWzrAJezDKtpqXl6Io8vQCZ04sPTn1N1QpOfPC70CWgLJVKuphVxOTJky0cUtMKLnOamB5TSt3CuOktCqgOjLBoKk0zl4ML9J6HVG5F/8tf8JTtRr5ef9zaqTStUDCngrA3/ewAzFFKPXhvg6ZZl4Mz9J6LVG7Jh3ZT2bt7O6eu3LZ2Kk0r8MypIJaKyBGMq5d+FxFvIM6ysTQtm+ydoPsUbBxd+dJ+Et+sO2ztRJpW4JnT1cYooAkQrJRKAG6T86uYNM1yXEth0/FL/OUEPgemcPxyjLUTaVqBZs4RBEAtoKeI9Ad6AI9ZLpKmPYQ63blXvROv2i5g/oo11k6jaQWaOX0xzQK+BJoBDU1T1rdoa5o1iODY5SsS7N3odPIDjp7P/JSZiNw3aZr2D3O6+w4GapsuRdW0/M/FC9Xhv9RbMpClCz6g+vCvrJ1I0wokc5qYDgClLR1E03KTa+ATHPZ6jHZXZ3Jy/1Zrx9G0AsmcCsILOKRHlNMKmnK9v+EmrtgvfRES9ZDqmpZd5jQxvWfpEJpmCe4lfVhS+x06H/4Pl1d8RKnOH1g7kqYVKGaNKAccAdxM02Fl3ohymmZ1rboMYCktKLnrazi/x9pxNK1AMecqpqeAvzA61nsK2C4iPSwdTNNyg7uTPRebvMcV5U7M0lFZr6BpWipzmpjeBhoqpS4DmO6kXgcssGQwTcstvVv5M3Tbm3jaVmWCtcNoWgFizklqm5TKweSqmetpWr7g6mhHSPPHWHI8UY8ZoWnZYM4X/SrTFUwDRGQAxvCgKy0bS9NyV/+mvrg72fFN+DFrR9G0AsOck9QjgO+B+hhjQUxWSr1p6WCalpvcnewZ8Igfqw9e4sjFW9aOo2kFgjknqf2AFUqp15VSr2EcUfhaOpim5baBj/ji4mDLN3q8CE0zizlNTL8AyWmeJ5nmaVqBUsLZgf5NfVm+/wLHL8emu0yySk53vqYVReZUEHZKqdTbUE2PHSwXSdMs57lmfjja2TAxPP2jiOkHp+dtIE3Lx8ypIKJFpHPKExHpAlyxXCRNsxwvV0f6Nq7E4r3nibx6m8DAwNSpVLVSfL3raw5cOWDtmJqWL5hTQQwFRovIGRE5A4wEBls2lqZZzpAWlbG1ESaGnyAiIiJ1Orr/KN7O3ry56U1uJ+ghSzXNnKuYTiilQoDaQB2lVFOl1AnLR9M0yyjl7kSvhhX4dVcU527cTZ1f3LE4nzX/jHOx5/hk+ydWTKhp+YPZN7wppWKVUnoMR61QGNKyCiIwacP9+zqBPoEMqT+EJSeWsOLkCiul07T8Qd8RrRVJ5UoU44nA8szbeZZLt+Lue21w/cE0KNWAD7d9SFRMlJUSapr16QpCK7JeaFWVpGTF9xtP3jffzsaOz5p/hiCM3DySJJVkpYSaZl3m3CjnLCLviMgU0/NqItLR8tE0zbIqlnSmS0BZfv4rkiux9+57raxrWcY0HcO+6H1MujyJY9d1Fx1a0WPOEcQ04B7QxPQ8CvjIYok0LQ+t+fhZTk15mXr+DQgKCrrvtXa+7RjdeDSR8ZH0WNqDMX+O4dLtS1ZKqml5z5zuvqsopXqKSG8ApdRdEREL59K0PHFwnzGI0EXT9KDeNXvjfs6dQ+6HmHNkDitPraRf7X4MrDsQVwfXPM2qaXnNnCOIeBEpBigAEamCcUSRYyLymogcFJEDIjJHRJxExE9EtovIMRGZJyL6bm0tz0VEXvvXPBdbF0Y0HMGSrksIrRjKlP1TeHzh48w5Moe7iXfTKUXTCgdzKoj3gFVABRGZDfyOcbNcjohIOWA4EKyUqgvYAr2AscD/lFLVgOvAczndhqbl1JBZuzh/I/0v/fJu5fm8xefMeXwOlYtX5pPtn9B8bnOGrRvGnCNzOBd7Lo/TapplmXOj3BqgOzAAmIPxxR7+kNu1A4qJiB3gDFwAWvPPKHUzgK4PuQ1Ny7a78YkMmRVBXELGVy7V9arL1LZTmdp2Kk9Wf5Izt87wyfZPaPdrO7ou6sq4nePYcXEHicmJeZhc03KfKKUyX0Dkd6VUWFbzsrVRkVeAj4G7wBrgFWCbUqqq6fUKwErTEcaD6w7G1NWHj49P0Ny5c3OUITY2FlfX/NGGrLNYL0doaOh9z8fNW8uEXfdoXMaWIfUdERGzslxOuMyBuwc4ePcgx+OOk0wyzjbO1ClWh3rF6lGzWE2K2RR7qKz55XcDOkt+zgFZZwkNDY1QSgVnWZBSKt0JcAI8gb2Ah+mxJ+ALHM5ovawmU1nrAW/AHlgE9AOOp1mmArA/q7KCgoJUToWHh+d43dyms/xbXuXAOLeWOiml1Ne/H1WVRi5TE8OP5yhLzL0Yteb0GjV682jVbE4zVXd6XRUwM0ANXjNY/Xz4Z3U+5nyOsuaX341SOkt68ksOpbLOAuxUZnxfZ3YV0xDgVaAsEAGkXLl0C/g2y5onY22AU0qpaAAR+Q1oCpQQETulVCJQHjj/ENvQtBx7MbQqhy/G8PnqI9Qo7Zrtu0ldHVx5tNKjPFrpUZKSk9gbvZcNZzcQfjacT7Z/wifbPyHIJ4iuVbvyWKXHcLZ3tsj70LSHleHfvlJqvFLKDxihlKqslPIzTf7AlIfY5hkgxHQDngBhwCEgHOhhWuYZYPFDbEPTckxE+KJHfWqVdueVOXu4EJvzQYRsbWwJ9Ank9eDXWdptKUu6LmF4g+FcuXuFd/58h9D5oYz5cwy7Lu1KOXrOUFava1puM+c+iAHAhAfmbQUCc7JBpdR2EVkA7AISgd3AZGA5MFdEPjLN+zEn5WtabnB2sGPKM8G0/2oTvx2Pp3culetX3I9B9QfxfL3n2X15N4uOL2L16dUsPL6QSu6VUo86rsVd4/q961yPu861uGvcuHeD+MR4mvzehJblW9KyfEt8XHxyKZWmpS/DCkJESgPlMK42asA/TUzuGFce5ZhS6l3g3QdmnwQaPUy5mpabypUoRs+GFZj6xyku34qjlLtTrpUtIgT6BBLoE8ioRqNYG7mWRccX8cP+H3CwccDDyQNPJ088nDyo6F4RD0cPIs9GcuLGCTZFbeJDPqSWZy1CK4TSskJLannWQt+/quW2zI4g2mIcPZQHxqWZfwsYbcFMmpZv9GlciSmbTzF3x1mGh1WzyDac7Z3pUrULXap2IT4pHnsb+3S/7Dfc2UDLli05ceMEG6I2sPHsRr7b+x0T906kuGNxfJx98CrmhVcxL7yLeRuPnb3wc/ejukd1XYFo2ZZhBaGUmgHMEJEnlFK/5mEmTcs3/LxcqFvSlp+3n+GFVlWws7VsB8gOtpl3ICAiVPWoSlWPqjxf73muxV1jc9Rm9kbvJfpuNFfuXOHkzZNcuXvlvvswKhevTKcqnXjc73HKuJax6HvQCg9zzkH8KSI/AmWVUu1FpDbQRCmlzxFoRULrinZM2B3H70cu07ZOaWvHuY+nk2fq0UdaySqZW/duEX03mj3Re1h2Yhnjd41nwq4JNCzdkI6VO/JopUd1f1JapsztzXU1xuWuAEcxLn/VtCLB39uWMsWd+GlbpLWjmM1GbCjhVIJqHtV4svqTzGg/gxXdVzAsYBgXbl9gzJYxqVdQXbydXjeFmmZeBeGllJoPJAOY7lPQI6hohcKDNwalx9ZG6NOoIpuPXeHUldt5nDD3VHCrwDD/YSzvtpxZ7WfRqUonlp9cTseFHZmwawKx8bHWjqjlM+ZUELdFpCT/9OYaAty0aCpNy2d6NqqAnY0wuwAdRWRERAgoFcCYJmNY2m0pbSq1Se2hdu6RuSQkJ1g7opZPmFNBvA4sAaqIyJ/ATOBli6bStHymlJsTbeuW5peIKO7GF54D6LKuZfms+WfMfXwulYtX5uPtH9N9cXfCz4TrG/O0zCsIEbHB6JOpJUZ3GEOAOkqpfXmQTdPylX4hlbh5N4Gl+wpfLzB1vOowte1UJoQa98QODx/OyE0jdSVRxGVaQSilkoH/KqUSlVIHlVIHlFL6+FMrkhr7eVKtlGuhaGZKj4gQWjGU37r8xqB6g1h5eiULji3IekWt0DKniWmNiDyhhxnVijoR4emQSuyNusm+qBvWjmMx9jb2vNTgJRqXacwXO74g8lbhrBC1rJl7DuIX4J6I3BKRGBG5ZeFcmpYnBg8efN+UlW6B5XB2sC1Ql7zmhI3Y8NEjH2FvY89bm9/SJ66LKHNGlHNTStkopRyUUu6m5+55EU7TLG3KlCn3TVlxd7KnS0A5Fu85z807hftLs7RLad5p8g77r+xnyr6H6cBZK6jM6jdARDxEpJGItEiZLB1M0/Krp0Mqci8xmV8izlo7isW1821Hx8odmbxvMvui9bUpRU2WFYSIPA9swrib+n3Tz/csG0vT8q86ZYsTWLEEs7efITm58F/lM7rxaEo5l+KtzW9xJ+GOteNoecicI4hXgIZApFIqFGgARFs0lablc/2aVOLUlduMXX2E45djrB3Hotwc3Pik2SecjTnL5zs+t3YcLQ+Z01lfnFIqTkQQEUel1BERqWHxZJqWj7WvW4aFu8/z/caTfL/xJFW8XWhXtzTt6pShbjn3Qte1dnDpYAbUHcC0A9NoWb4lQuF6f1r6zKkgokSkBLAIWCsi19HjRWtFnJO9LTMHNuLizTjWHLrIqgMXmbTxJN+Gn6BciWK0q1ual0Kr4uGSeffdBclLAS+x9fxW3tv6Hm94vWHtOFoeyLKCUEp1Mz18T0TCgeLAKoum0rQConRxJ/o38aV/E1+u3Y5n3eFLrD5wkRlbTrMz8jo/P98YF0dz9sPyPwdbBz5t9ik9l/Vk0uVJBMYEUt6tvLVjaRZkzklqz5QJ2A/8ganjPk3T/uHp4sBTwRX4cUBDvns6iAPnbjL0pwjiE5OtHS3XVPWoyhctvyA6IZqnlj7FmtNrrB1JsyBzTlLvwjgpfRQ4Znp8SkR2iUiQJcNpWkH1aG0fPu1ej83HrvD6/D0kFaKrnVpXbM3IMiPxLe7LGxvf4MOtHxKXGGftWJoFmFNBrAI6KKW8lFIlgfbAfOAFYKIlw2laQfZUcAXeal+TZfsu8P7Sg4Wq4zsvey9mtJvBgDoDmH90Pn1W9OHkjZPWjqXlMnMqiGCl1OqUJ0qpNUALpdQ2wNFiyTStEBjSsgpDWlRm5tZIxv9+zNpxcpW9rT1vBL/BxLCJXLlzhV7Le7Hw2MJCVREWdeacPbsmIiOBuabnPYHrImKLaZQ5TdMyNqp9Ta7ejuerdcco6eJAvya+1o6Uq5qXb86Czgt4a/NbjNkyhpmHZuLv7U9dr7rU86pHlRJVsLP591fNjbgbnLp1ilM3TxEVE0W3qt2o4F7BCu9Ay4g5FUQf4F2My1wF4yR1H8AWeMpy0TStcBARPutejxt3Ehiz5CAeLg50rF826xULkFLOpZj86GTm/j2Xzec2s+7MOn499isATrZO1CpZizol63A38S6nbhqVwvV71+8rY2PURmZ3mI2TnZM13oKWDnMuc71CxiPIHc/dOJpWONnZ2vBNnwb0//EvXpu3h1JuTjTy87R2rFxla2NL31p96VurL0opomKi2H9lP/uv7Ofg1YP8cvQXXOxd8HX3pXXF1vgV9zMmdz9O3jzJS+tf4sudX/J/If9n7beimWRYQYjIV0qpV0VkKelc1qqU6mzRZJqWB3bu3Jln23Kyt2XKM8F0+/ZPXpi9i+XDm+HjXjj3lkWECu4VqOBegQ6VOwCglMrwDvMK7hV4pvYzzDg0g5AyIbSp1CYv42oZyOwIYpbp55d5EUTTrCEoKG+v1C5ezJ5J/YLoaqok5gwKwcHOrE6VC7ysuh95JfAVdl7ayZgtY6hdsjZlXQtXM1xBlOFfplIqwvRwJ7BZKbVRKbUR4xzEjrwIp2mFUXUfN8Y+UZ+IyOt8suKwtePkG/a29nzR4guSVTJvbnpTD1KUD5iz6/I74JzmeTFg3cNsVERKiMgCETkiIodFpInpbu21InLM9NPjYbahaflZJ/+yPNfMj+lbTrN4zzlrx8k3KrhX4N0m77I3ei/f7fnO2nGKPHMqCCelVGzKE9Nj50yWN8d4YJVSqibgDxwGRgG/K6WqYVRKox5yG5qWr41qX5NGvp6M+nU/Ry7qUXxTtPdrT/dq3flh/w9sPb/V2nGKNHMqiNsiEpjyxNS9xt2cblBE3IEWwI8ASql4pdQNoAsww7TYDKBrTrehaQWBva0N3/RtgJuTHUNnRXDzrm5SSTGq0Sj8ivsx+o/RXLl7xdpxiizJ6q5HEWmIcZNcShffZYCeac5RZG+DIgHAZOAQxtFDBMagROeUUiXSLHddKfWvZiYRGQwMBvDx8QmaO3fug4uYJTY2FldX1xytm9t0FuvlWLp06X3PO3XqlOdZjl5PYuxfcdTzsmV4oCM2GZzMzS+/G8ibLOfjz/PlxS+p4liFYaWGYSPp78/ml88lv+SArLOEhoZGKKWCsyxIKZXlBNgDdYF6gL0562RSVjCQCDQ2PR8PfAjceGC561mVFRQUpHIqPDw8x+vmNp3l3/IqB8Yl3KmTtbJM/eOkqjRymfpm/bEMl8kvvxul8i7LvCPzVN3pdVXr+a3Vx9s+VtvPb1cJSQlWyZKV/JJDqayzADuVGd/XZnVUr5RKAA6Ys6wZooAopdR20/MFGOcbLolIGaXUBREpA1zOpe1pWr43oKkvu8/c4Ms1f3Ph5l3eeLRGoRpsKKeerP4kxR2Ls/LUShYeW8icI3PwcPQgtGIoYRXDCCkTYu2IhVqej2SilLooImdFpIZS6m8gDKO56RDwDPCZ6efivM6madYiIox9oj6eLg7M2hbJsn0XeOOxGvRpVBFbm6I7vKeI0Na3LW1923In4Q5bzm9hbeRa1pxew2/HfsPV3pW6DnUpdqEYDUs3zLAZSssZaw119TIwW0QcgJPAsxgnzOeLyHPAGeBJK2XTNKso5mDLe53r0KtRBd5fcoh3Fh3g5+1neK9TbRpXLmnteFbnbO9Mm0ptaFOpDfFJ8Wy/sJ3Vp1ez+uRqnl/zPD7OPnSo3IGOlTtS3aO6teMWCmZVEKZ7EqoBqf0CKKU25XSjSqk9GOciHhSW0zI1rbCoWdqdnwc1ZuWBi3y8/DA9J2+jk39ZWpXQnSencLB1oHn55jQv35wW8S1I9ktm2cllzDo4i2kHplHDowadqnSie7XuuDm4WTtugZVlBSEiz2NcZVQe2AOEAFuB1paNpmlFl4jQoV4ZQmuUYtLGE0zaeILVycn8rQ4zuEVlvFz1UCwpHGwcaOXXinZ+7bgWd41Vp1ax/ORyvtz5Jd/v/Z7etXrzdK2n8XDS995mlzkNdq8ADYFIpVQo0ABj2FFN0yysmIMtrz1anXWvtyS4tB0/bD5J87HhfLbyCNdux1s7Xr7j6eRJn1p9mP34bOZ3nE9I2RAm75tM21/b8t+d/9X3VGSTORVEnFIqDkBEHJVSR4Aalo2laVpaFTydGVzfkbWvt+SxOj58v+kEzceu54vVR7hxR1cU6alVshbjWo1jYeeFtK7YmpmHZtLu13Z8uv1TLt6+aO14BYI5FUSUiJTAGDBorYgs5p+b5jRNy0NVvF0Z36sBa15tQWjNUkzccIJmY8P5ePkhVh24yJmrd/SQnw+o6lGVz5p/xtKuS3m88uPM/3s+HRd2ZOfFvOvqvaAyZ8CgbqaH74lIOFAcWGnRVJqmZaqajxvf9Ank5YsxjP/9KD/+cYopm08B4OZoR60y7tQq40btsu4EVfKgail9oraie0Xeb/o+Q+oPYei6oQxfP5zp7afrK54yYc5J6llKqX4AyujuGxGZBfSzcDZN07JQo7QbE/sGEZeQxN8XYzh04RaHzt/i8IVbLIiI4vbWJAAaVCxBn0YV6Vi/LMUcbK2c2rrKupZlUptJ9FvRj2HrhvFT+58o41rG2rHyJXMuc62T9omI2AJ5O8qKpmmZcrK3xb9CCfwrpHZnRnKyIvLaHdYfuczP2yMZsWAfHyw7RLcG5ejdqCK1yrhbMbF1lXUty3ePfseAlQMYum4oM9vPpLhjcWvHyncyPAchIm+JSAxQX0RumaYYjC4w9F3OmpbP2dgIfl4uPNfMj3Wvt2T+kCa0qeXD3B1naT9+M12//ZN1hy5ZO6bVVPeozvjW4zkbc5aXfn+JuMQ4a0fKdzIbUe5TpZQb8IVSyt00uSmlSiql3srDjJqmPSQRoZGfJ//rGcBfo8MY07E2t+4mMOSnCCIir1s7ntU0LN2Qz5p/xt7ovYzYNILE5ERrR8pXMmxiEpGapktaf0k7HkQKpdQuiybTtDzw/fffWztCnivh7MDAZn70CC7P4xM2M3zOblYMb05xZ3trR7OKx3wfY9TdUXz616d8vP1jxoSMyXL87KIis3MQr2OMu/DfdF5T6DuptUJg8ODB1o5gNe5O9nzdO5Ae321h1G/7mNg3sMh+Mfap1Yfou9H8sP8HSjqV5MWAF4vsZ5FWhhWEUmqw6Wdo3sXRNC0vBVQowZvtavDJiiP8tP0M/UIqWTuS1QxvMJzoO9F8v+97Dl49yLtN3qW0S2lrx7KqLG+UExEnEXldRH4TkV9F5FURccpqPU3TCobnm1WmVQ1vPlx2iEPni+7Y2CLCB498wKhGo4i4FEHXxV1ZcHRBkb7x0Jw7qWdiXOr6NfANUBuYZclQmqblHRsb4csn/SlRzJ6X5uziTnzRPVFrIzb0rdWXXzv/Sp2SdXh/6/sMWjuIc7HnrB3NKsypIGoopZ5TSoWbpsGAvvVQ0woRL1dHvuoZwKkrtxmz+KC141hdBbcKTHlsCu+EvMP+6P10W9yNOUfmkKyKVpfr5lQQu0UkdVw/EWkM/Gm5SJqWdyIiIu6birKmVb14ObQqCyKiWLg7ytpxrM5GbHiqxlMs6rKIBqUa8Mn2T3hh3QtF6lJYc+6kbgz0F5EzpucVgcMish9jkPf6FkunaRYWHHz/uFVFub0ZYHhYNbaevMr/LTxAQAUP/LxcrB3J6sq4lmFSm0nMPjybsTvGMv3gdJ6v97y1Y+UJc44g2gF+QEvT5Ad0ADoCnSwXTdO0vGZna8P4Xg2ws7Vh+JzdJCQVrSaVjIgIfWv15dFKjzJxz0SOXz9u7Uh54v/bO/PwqnLYjcsAABn3SURBVMprcb8rCUkIAUlCEuYIZZLBUEZBsQQEZEasOF+c21tvrVpqW4tVi1fU2kHrePvTotaKUq0CCioQB7RVARkFZVAGQUEEIcjM+v3x7cgxhuQQsgdO1vs8+8nZ++yT/Z5zvrPX/tY37EoDhKquVdW1wG7c+Ad1m7/ZbhhGAtG4fm0mju7Ekk+/4uHXV4etExlEhN/0/A2ZtTK56a2bakSqKZ5uriNEZCXwMfA68Ak23bdhJDRDOjVi6MmNuGf2SpZvqrldX8uSUzuHG0+5kaVblzJp2aSwdXwnnhTTBNx9qD9S1RZAf6yR2jASngkjO3JC7VqMm7LIUk0xDCoYVGNSTfEEiP2quhVIEpEkVS0GOvvsZRhGyGTXSeW2UZ1YtnEHDxRbqqmUmpRqiidAbBeRTOAN4EkRuQdI3E/EMIxvOLNjQ0YUNuYvc1aybONXYetEhpqSaoonQIwEvgauA2YCq7HeS4ZRY7h1RAfqZ6Qybspi9h2wVFMpNSHVFE+AyANSVfWAqj4G/BWwG9waRg0hq04qt5/VkeWbdnB/cWKeCKtC2VTTQT0YtlK1E0+AmALEXjYc9LYZhlFDGNihIWd9vwn3F69i6aeWaiolNtX0/LbnWbdjXUINtoxnJHWKqu4rXVHVfSKS6qOTYRgR5Obh7Xlr1ReMm7KIcScnzknwWBlUMIjiFsW89PFLvPav18hOz6Zzbmc657mlfU570pLTwtasEvEEiC0iMkJVpwKIyEjgC3+1DMOIGvUzUpk4uhOXPzaPZ9JS6F+kdlMdXKppYp+JFO4pJKUghUVbFrFw80LmrJ8DQEpSCl3yujCs5TAGFAwgMzUzZOP4iSdA/BjXe+k+b30DcPGxHlhEkoF5wKeqOkxEWgCTgWxgAXBxbM3FMIzw6X9SPpf0PpFJb3/CbS8uZ/zQkyxI4Cb2a5zamL5t+zKm7RgAtu7eyuIti3l/y/vMXjub3779W25/53aKmhcxvOVwejXuRUpSPKfg8KjUTlVXA6d4XV1FVXdW07F/BiwH6nnrdwJ/UtXJIvIQcDnwYDUdyzCMauLm4e359NMNPDL3Y/bsP8iEkR1JSrIgUZac2jkUNS+iqHkR13W5jsVfLGba6mnM+HgGMz6eQU56DkNaDmF0q9G0ymoVtm65xB2+VLWkug4qIk2BocD/AteLuwTpB1zg7fIYcAsWIAyfufLKK8NWOO4QES5ol0qrFgU8+Npqdu8/yF1nn0xKcjx9XmomIkJhbiGFuYXc0P0G3tzwJtPWTOOpFU/xxAdP0KdJHy7teCnd8rtFqkYmYbS4i8g/gYm47rLjgEuA/6hqK+/5ZsAMVe1YzmuvAq4CyM/P7zp58uQqOZSUlJCZGY1coLlE1wOi4xIVD3AuderUYdqa/Ty3cj/dGybzo5PTSAmhJhGVz6UqHiUHS5i7cy6v73ydkkMlNE9tzhn1zqAwo5AkqXrArcylqKhovqp2O+IOpajqERdcN9jeFe1ztAtumvAHvMd9gelALrAqZp9mwJLK/lfXrl21qhQXF1f5tdWNuXyXqHioRsclKh6q33b56xurteCX0/Wyv72ru/cdCNUlTI7FY/f+3fr0iqd18LODteOkjjr42cE6eflk/Xr/1764APM0jvN1hSFKVQ8Bf6g0yhwdpwIjROQTXKN0P+DPQH0RKU15NQU2VvNxDcPwgSv6tGTCqI7MXrGZKx6bV6PvaV1V0lPSGdN2DNNGTeOPff9I/bT63PbObfSf0p+J70xk5baVoXjFU4d5RUTOlmpKjKnqr1W1qaqeCJwHzFHVC4Fi4IfebmOBF6rjeIZh+M/FpxRw9zmFvL36C4beO5fiDzeHrXRckpyUzICCATw55EkmnTmJPk36MOWjKYyeOpqLX7qYF1a9wJ4DewLziSdAXI8bOb1PRHaIyE4R8WOC+F/iGqxXATnAIz4cwzAMn/hh16Y8fllPBLj0b+9x+aT3+OSLXWFrHZeICF3zu3Ln6Xcy+5zZjOs2ju17tzP+rfH0m9KPO969g9Xb/Z9hN547ytVV1SRVraWq9bz1epW9Lh5U9TVVHeY9XqOqPVS1laqeo6p7q+MYhmEEx2mtGzDz2tO5cUg7/rNmKwP/9AZ3zlzBrr2WdqoqWelZjO0wlqmjpvLooEc5rclpPPPhM0xdPdX3Y1fazdVLLV0ItFDVCV4Po0aq+q7vdobhM2Uzp5pA8+iERWpKEled/j1GdW7CnTM/5MHXVvPcgg38anA7RnVuEqlunMcTIkL3ht3p3rA72/ZsQ/G/rMaTYnoA6MXhMQolwP2+GRmGkRDk1UvnD2MKee4nvcmvl851Ty/ilImz+dWzi3l52WdWqzgGstKzyE7P9v048QyU66mqXUTkfQBV3WaT9RmGES9dmmfx/E9OZfqSTcxcuonpizcx+b31pCYn0bNlNkVt8+jXLo8TG9QJW9UoQzwBYr83b5ICiEgu357+2zAMo0KSkoQRhY0ZUdiYfQcOMe+TL5mzYjNzPtzM76Z/wO+mf0CX5vW55NQWDO7YkFo2KjsSxBMg7gX+BeSJyP/iuqKO99XKMIyEJTUlid6tGtC7VQPGD2vP2q27eGXZ5/z9nbVc89T75NVN48KeBZzfsxl5ddPD1q3RxDNZ35MiMh/oDwgwSlWX+25mGEaNoCCnDlee3pLLT2vB6x9tYdLbn/CnWR9xX/FKhnRqxH/1KqBpVka5rxVg9wHl0CG1CQN94IgBQkTScVN9twKWAA+rqrUqGYbhC0lJQlG7PIra5bFmSwmP/3st/5y/gRcWxjGpwqyXyExLoU5aMplpKWSm1yI7oxa9vpdDv3Z5fC8303pPVYGKahCPAfuBN4HBwEnAtUFIGYZRs2mZm8ktIzowblBbZi//nF17y7/f8yFVliz/iPymBZTsOcCuvQco2XuAnXsP8On23dz+0gpuf2kFzbJrU9TWBZ9eLXNIr5Uc8Ds6PqkoQLRX1U4AIvIIYOMeDMMIlMy0FEZ2blLhPq/t+Zi+fduU+9zG7bsp/nAzxSs2M2XeBh7/91rSayVxWqtcxnRrSr92eTZNeQVUFCD2lz5Q1QNWPTMM43ijcf3aXNizgAt7FrBn/0H+s2YrxSs2M2PpZ8xa/jn59dI4t1szxnRvdsR2jppMRQGiMGbOJQFqe+sCaHVNt2EYhhEE6bWS6ds2j75t87hpWHvmrNjMU++u4y/Fq/hL8Sp+0CaXC3o0t1pFDEcMEKpqSTrDMBKSlOQkBnZoyMAODdmw7WueeW89T89bz1VPzCe/XhpjujXjXKtVxH/LUcMwjESkaVYG1w9syzX9W39Tq7iveBX3ebWK83s0p38NrVVYgDAMw6CcWsW8DTz93jp+9MR88uqmcW73ZpzXozlN6tcOWzUwLEAYNZouXbqErWBEkKZZGVw/oA3X9GtF8Ydb+Mc7a7mveBUPv7GGnw9owxV9WpJcAwbmWYAwajTz588PW8GIMCnJSQxon8+A9vls2PY1E6Z/wMQZK5ix9DPuPudkWuXVDVvRV2peUs0wDKMKNM3K4KGLunLPeZ35ZOsuhtw7l4deX83BQ4l7DxELEIZhGHEiIozs3IRXr/sBRW1zuWPGCs5+8G1Wbd4ZtpovWIrJMAzjKMmtm8ZDF3Vl2uJN3PzCUobcO5fTGiexnNXk1Eklq04q2aVLRir1aqccl3NBWYAwDMOoAiLuHhe9WuZwy7RlvLx0E3PWrSh333rpKRQ2q09nbylsVp8GmWkBGx89FiAMwzCOgdy6adx/QReKi4vpeWoftpbsY9vX+/hy1+Fl9ZZdLFy/nfuLV1HaZNE0qzadm9Xn1FYNOLNDQ7LqRO9GnRYgjBpN165dv7VuvZqMqiIiZKSmkJGdQrPs8kdgf73vAEs/3cGi9dtZuH47C9ZuY/riTdz0/FJOb5PLiMLGnNE+n8y0aJyao2FhGCGxYMGCsBWMGkRGago9WmTTo0U2AKrKso07mLZoI9MWbWTOis2k10qif7t8hhc2om/bvFCnJrcAYRiGERIiQscmJ9CxyQn88sx2LFi3jamLNvLi4k28uGQTGanJFLXNY1DHhhS1zaVueq1A/SxAGIZhRICkJKHbidl0OzGb3w5rz7/XbGXG0s94ZdnnvLhkE6nJSZzW2rVXnNE+n+wA2iwsQBiGYUSMlOQk+rTOpU/rXCaM7MiCddt4eelnzFz2GXNWbCbpOfhpv9ZcN6D8GyVVm4ev/90wDMM4JpKThO4nZtP9xGx+M/Qklm3cwcvLPqNz8/q+H9sChGEYxnFCbJtFEAQ+1YaINBORYhFZLiLLRORn3vZsEXlVRFZ6f7OCdjMMwzAOE8ZcTAeAn6vqScApwNUi0h74FTBbVVsDs711wzAMIyQCDxCquklVF3iPdwLLgSbASOAxb7fHgFFBuxmGYRiHEdXwpqoVkROBN4COwDpVrR/z3DZV/U6aSUSuAq4CyM/P7zp58uQqHbukpITMzMwqvba6MZfwPIqKir61XlxcHJpLZUTFA8wlyh5QuUtRUdF8Ve1W6T9S1VAWIBOYD4z21reXeX5bZf+ja9euWlWKi4ur/Nrqxly+S1AewLeWMF0qIyoequZSHlHxUK3cBZincZynQ7kfhIjUAp4FnlTV57zNn4tII+/5RsDmMNwMwzAMRxi9mAR4BFiuqn+MeWoqMNZ7PBZ4IWg3wzAM4zBhjIM4FbgYWCIiC71tNwJ3AM+IyOXAOuCcENwMwzAMj1AbqY8VEdkCrK3iyxsAX1SjzrFgLt8lKh4QHZeoeIC5lEdUPKBylwJVza3snxzXAeJYEJF5Gk8rfgCYS3Q9IDouUfEAc4myB1SfSyiN1IZhGEb0sQBhGIZhlEtNDhD/F7ZADObyXaLiAdFxiYoHmEt5RMUDqsmlxrZBGIZhGBVTk2sQhmEYRgVYgDAMwzDKxQKEYRgJhYhE5rzmzRxx3BKZDzJKhPmlHu8FqiYR9ncV9vFLEZEMEUmLgEcPEWmgqoci4JIjInX0OG/ktQABiEgXERkpIgUikqGqGsZViIgMByZE4YcvIu1EpI+IpIlIirctcC8RGS4iNwR93PKISjnxXCJRVkRkNPB3YIaIDBORliF5DMTN8dYkjOOXcRkNPAW8KCJXikjPsJ2qSo3vxSQiI4C7gYXA18BO4HZV3SQiSUFdjYjIINx8VL9Q1VllnpMgr0S8An4H8DFuuP58YJKqfhmki/ejvxO4QVVfDeKYFbhEopx4LpEoKyLSAngZuBBoC/TCzcL8gqourOi11ewxArgFuEJVF4hIsqoeDOr4ZVwaA8XA+bjpLroBzYFnwy7DVcFqEDAE90MbA9wLbAP+LCINAwwOhbh+y7eq6iwRyRKR73tXqnW8K9VArhS92sLZwOWqOgiYAuQDvxCR7ABPPqcATwI/VtVXRaS+iLQQkbqlNZqAGUzI5QS+KSsPE4GyAtQDNqjqe6r6d+BvuAlAh4tIQUAOAD8EmnjBoR5wl4g8LiJneSfsIEnG3fxsgaq+AkwGFgFniUjXgF2OmRobIEQk2XuYBJwEoO5WqH8FVuBOiLUD0tkMrAYaiEhnYBruiugPwG9EpF6ANYgk4ASgE4CqPo+bih3gggBTKpuBXUBLEWmAm/79PtztaC8JMuftvedkwi8n4Gp0a4DcsMuKqi4CtovIT731ebiy0hxoF4SDd9z/AuaKyArc57EOeA93oVNU0Wt9cFkPfCkid3vra4BXcOW5E0Sn7Sgu4rmrUKItwADgEu9xF1za4BxvXYBTgCeAhgF4XO49/h6uur4auNLbdjouv9spgM8kH2gQc9ypQP+Yz+Rc4GkgJQCPfO9xW2AlsCnmM7kYeMbv78Y7Vi0g1XvcGXgfGBN0OYlxSfcetwBm4gJFoGUF6AuMAS721kfhUm/nxuxzGS6g1wrAY2zMtqdwab/S9fO9cux3mW0KnBCz3hl4ABgXs22o952l+11WqvW9hS0Q+BuGM4CtuDxyW2/bKODF0h+/t20aMCBgj2bARWX2+6efHt4xRgKvAe8AF3jbfgL8v9hjA7OAbgF5lJ6AWuHSTLH7veinR4zLk97nPxhI9X7k08ucDH0tJ+W5eNsaln5XQZUV3NX4JmAcrl3qj7ha1dW4GszPvf3O93xTA/CYh6vN1Y95vrRt9UJc0PTFwzvGKFxN8nog19uWjktdPwjc42071ysrdfwsK9X+/sIWCPTNuh/4QtyV3y+An3nb6wIjgCXAeOBaYDnQLCgPyrnKwVWR3/fLwztGd8+lo+f1JpAG5AJXeSfja4FLvc/El6vlcjzmAlnl7PdD7zPx7aod17C4EFe7HAa8DtzsnQyHAIuDKCdHcHkDuA2vthdUWcHVmO4CrvPW0/Gu2IE23ufyPDAb+AD4foAeT+LSj3kx+13pBQ/falTeb2Q2ru1lAvDTMkGiA67W/ZJ3bvHlM/FzCV0gsDfqUheTgL4xBWgWkBSzTwdc75C7gcIAPV4t9eDw1c+luCuTDj5/LmcDj3mPTwD+g6s5XIjLmfbAXYX91c8CXo7Hv3HdFi8DWnjbL/FOyEF8JpNi1v/bO/le5q2fjOtd5Vs5qcDlx57LFbj2oiTvMwqirJwHPMThFGAGLt33p5h9WgM5IXhMBh7w1ht7z/udbkv1fiNp3vd0jxckGpbZL5+YFNTxtIQuENgbdb0rTiizbSbwe++xhOxxV5ltrYHWAfh0AP4FPIrLaY/HVZsnAdfEOCeF4DHS87ja26cPXjrOZ5dunscAb/1Gb30u8IMgykklLo94Lqd629r4VVZwac80oDZQgLtaHwDU9p7PwKWbzvL5c6jMo7bnMRRXy0jz0aW5Fxwyymw/G9fD7ael312QZcWPJYzugoEiIj1wBWuXut4niEhtVd2Nuwo8V0SyVHWbn33Ij9IjWVVX+uER41LLc1koIjcB38dd+d3m7bMDuElEJqnqjhA9dgLjReRxVX3TD48yLiWqOk9ElgNXich1gKrqUBG5CuiHSzn5xlG4DATeUtWPfPIYiiubb+PSsNfj0krXuqdlibpxILOBfX44VMGj9De8NwCXE0TkZlVdAaCqz3odlApF5Hmgv4i0VdWNfrgEQUIHCO/LvB2XR28qIrtV9UfeSRngQ9wJ6TzgQR+Dw9F6+DbIp4xLcxH5SlWvBpaKSHcR6a+qs4FMXAO6L338j9JjN+Bb180yLgUisklVx4lIXVzvsmXerlnAAb88quDiSznxumE2xaVb/weX1huL6zzQC9f4epG376fABbi0ZEJ6VOByETBHRAao6jL4JkichevJ1Pt4Dg5A4qaYcFXfGRzuqtkc1xf50TL7nY3rXpqBD2mmqHhU4vI3b/0G3JXZ88AC/GuHiYRHBS5bgUfK7HcNsBQ4qYa4JOMGbzbhcLvYdcBaoBEuzz8S+C0+pv2i4lGByzXAp0Abb70RrpG+s58uQS2hC/j4ZdbBNaB1jNn2e9wV2B9itp0AZCe6RyUuH+AGWyXhct5n4TUMJ7JHvN8PLr99i5+BKiouuG7F3YEcXA+cG8o8/2tcm4hvOf4oecTpcgOuray0PSTTb6egloQbSS0ibQBUdRfuhDNTRM4RkYdxP67hQD0Rqe/t95WqfpmoHnG6DMNdrdZW1Xmq+i9V/ThRPeJ0+eb7UZcKvFXdyOGEdRGRYcBzuN5Zt+Iagn8iIr+O2e0pXJrNzzaHSHgchcsznsceAFUt8dMpSBIqQHhf5kIReRpAVW/BXYG1Bbbj+k6vAfJwP7yE9jhKlxzc3DoJ7XGULt98P+pdGiaqi4j0xp0Ex6rqD3C9dHoAvYH/FpHxItIKN4K5C1C/uh2i5BE1l9AIuwpTXQuuej4TN7hrEvDUEfa7CNdFsEEie0TJJSoe5lKhS2+86We89VzgRe9xS1w65wH8H3wWCY+ouYS1hC5QzV9oY1yvlwa4aQf+EfNcCnAm8C4+NyBFxSNKLlHxMJcjeiQD9WIeN8UNyGvkbSugnDE8ieoRNZewloRKManqRlUtUdUvgB8BaSLyd+/pdrgq4gj1ea76qHhEySUqHuZyRI+Deni8i+DSW1+qG19wEW6AXi1V/aomeETNJSwS+oZB4qaJ/j2uqpiEGwUbeL/kqHhEySUqHuZSocsk3KR4A3GpliU12SNqLkGQ0APlVPULEVmMm41zQFg/tKh4RMklKh7m8l28QWG1cFOb1MKNy/BtZH/UPaLmEiQJHSBEJAs3y+TAkK86IuERJZeoeJjLd1GXVtgnIhOA98I6EUbFI2ouQZLQKSYAEUlX1T3mcZiouETFA8zlCB6B3gs96h4QLZcgSPgAYRiGYVSNhOrFZBiGYVQfFiAMwzCMcrEAYRiGYZSLBQjDMAyjXCxAGEaciGOuiAyO2TZGRGaG6WUYfmG9mAzjKBCRjsAU3B0Ak4GFwJmquvoY/meKqvp6pzrDqAoWIAzjKBGRu4BduNlYd6rqBBEZC1yNmzvpbeB/VPWQiPwfbiro2sDTqvo7739sAB7GTcb3Z1WdEsJbMYwKSeiR1IbhE7fiboW6D+jm1SrOwt2D+IAXFM4D/gH8SlW/FJEUoFhE/qmqH3j/Z5eqnhrGGzCMeLAAYRhHiaru8m7wU6Kqe0XkDNwtKee5KXuoDaz3dj9fRC7H/dYaA+1xd40Dd/tKw4gsFiAMo2oc8hZwU0E/qqo3xe4gIq2BnwE9VHW7N413eswuuwIxNYwqYr2YDOPYmQWM8abqRkRyRKQ57tapO4EdItIIGBSio2EcNVaDMIxjRFWXiMitwCwRSQL2Az/G3YryA2ApsAZ4KzxLwzh6rBeTYRiGUS6WYjIMwzDKxQKEYRiGUS4WIAzDMIxysQBhGIZhlIsFCMMwDKNcLEAYhmEY5WIBwjAMwyiX/w9ciZXJ+pP69AAAAABJRU5ErkJggg==
" style="border: 0px; box-sizing: border-box; height: auto; max-width: 100%; vertical-align: middle;" /></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
Vishal Misrahttp://www.blogger.com/profile/04748385949397825305noreply@blogger.com4tag:blogger.com,1999:blog-4160882474275537188.post-42076182041693495792019-08-25T04:37:00.002-07:002019-08-25T04:59:09.000-07:00Who is paying for the Tariffs? China or the US Consumer?<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<h2 style="text-align: justify;">
Who is paying for the Tariffs? China or the US Consumer?</h2>
<div>
<br /></div>
<div>
<br /></div>
<div>
We present a counterfactual (i.e., what <i>might have been</i> if the tariffs were not in place) analysis of the impact of tariffs imposed on China by the Trump administration, starting in February 2018. If you want to skip straight to the results, scroll down to the charts and the green curve towards the end represents the estimated price of the goods in the absence of tariffs. The maximum percentage difference between our estimate and the actual price is also marked in the charts.</div>
<div>
<br /></div>
<h4 style="text-align: left;">
Brief background of the Synthetic Control Method</h4>
<div>
<br /></div>
<div>
The analysis is done using the Synthetic Controls method pioneered by Abadie et. al., and which I have recently started extending with colleagues at MIT. A more detailed explanation of the technique can be found <a href="http://peerunreviewed.blogspot.com/2019/05/mrsc-new-way-to-answer-what-ifs-and-do.html?m=1">here</a>, but the basic idea is to analyze the impact of a "treatment unit" by creating a synthetic version of the treatment unit using a combination of untreated units, and then compare the behavior of the treatment unit post the intervention with the behavior of the synthetic version, which has not undergone treatment, i.e. the "counterfactual". The reason you can build a model of the treatment unit from other units is the assumption that there is correlation between those units. For instance the price of one commodity is affected by factors like gas prices, seasonal demand, consumer confidence etc., and those same factors (or some combination) affect prices of other commodities. </div>
<div>
<br /></div>
<div>
In our analysis here, the "intervention" is the application of tariffs in February 2018, shown by the vertical line in all the graphs, and the price data is obtained from the <a href="https://www.bls.gov/data/">Bureau of Labor Statistics</a>. The data is from January 2010 to July 2019 and includes 328 separate categories. The prices are relative prices with the 1984 price of the corresponding good set as 100. The categories of tariff affected and unaffected goods were obtained from Anthony DeBarros and Josh Zumbrun as a follow up to their WSJ article <a href="https://www.wsj.com/articles/despite-reprieve-new-tariffs-will-hit-wide-range-of-consumer-goods-11565960400">Despite Reprieve, New Tariffs Will Hit Wide Range of Consumer Goods</a>. The counterfactual curve in all the plots is our estimate of what the <b>price would have been if the tariffs had not been imposed.</b></div>
<div>
<br /></div>
<div>
<br /></div>
<h4 style="text-align: left;">
Initial findings</h4>
<div>
Our initial analysis indicates a clear rise in prices paid by consumers as a result of tariffs. We also present a few counterfactuals of tariff unaffected goods, as an example of placebos where no such clear trend of rise in prices can be observed. </div>
<div>
<br /></div>
<h4 style="text-align: left;">
Caveats: </h4>
<div>
<ol style="text-align: left;">
<li>This is a very quick and initial analysis of data, for a detailed economic model and analysis please look at the paper <a href="https://www.nber.org/papers/w25672.pdf">The Impact of the 2018 Trade War on U.S. Prices and Welfare</a> by Amiti, Redding and Weinstein. Our analysis is generally supportive of their conclusions.</li>
<li>Our analysis does not study and draws no conclusions on the long term impact of tariffs.</li>
</ol>
</div>
<h2 style="text-align: center;">
</h2>
<h3 style="text-align: center;">
Tariff Affected Goods Counterfactuals</h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi64uclBGSvQzlF0hkSRfGy4_sfeaaVsF6FxHAd7qhtmjApwJt5XN2hrCD5scuDexd_s4SaQUN6GQHLJO3Pwh6dOFRJ6fKUhd5imovTlkm4GeELs8q5rG0FrHRHdcocGmWPfXfAPYyMZTo/s1600/Appliances_tariff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="288" data-original-width="432" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi64uclBGSvQzlF0hkSRfGy4_sfeaaVsF6FxHAd7qhtmjApwJt5XN2hrCD5scuDexd_s4SaQUN6GQHLJO3Pwh6dOFRJ6fKUhd5imovTlkm4GeELs8q5rG0FrHRHdcocGmWPfXfAPYyMZTo/s640/Appliances_tariff.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgM-dT80_c-335hqoCijXOAv0EqGlSZxBtYMhuJX9rdnEOKin-D4RluPLrGvWm6Vi_5NP9-JIPdqYbZVv6y62KzCKJVXAbO03tFpNyds3sYWAcuqwuqnEiIx8wPh7AYoGM3GEFMyfWEsOs/s1600/Floor+coverings_tariff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="288" data-original-width="432" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgM-dT80_c-335hqoCijXOAv0EqGlSZxBtYMhuJX9rdnEOKin-D4RluPLrGvWm6Vi_5NP9-JIPdqYbZVv6y62KzCKJVXAbO03tFpNyds3sYWAcuqwuqnEiIx8wPh7AYoGM3GEFMyfWEsOs/s640/Floor+coverings_tariff.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlDGYlgDUVJtHaAsJqKt_lm1t0OIJzZrjDAOAThyphenhyphen8g6ovXZ_qz6PsqI2EDElnj8FCktihf_5GnSlQR-fWPjf3SKDw40RG627dwrlElIHv5fSAVCEPXMr90DVFyLQ5Mq2QiLToxkODkTSI/s1600/Laundry+equipment_tariff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="288" data-original-width="432" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjlDGYlgDUVJtHaAsJqKt_lm1t0OIJzZrjDAOAThyphenhyphen8g6ovXZ_qz6PsqI2EDElnj8FCktihf_5GnSlQR-fWPjf3SKDw40RG627dwrlElIHv5fSAVCEPXMr90DVFyLQ5Mq2QiLToxkODkTSI/s640/Laundry+equipment_tariff.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVNQ5ny5XcXm6UhFqdRikie6uubu-wL9n416EU_4QnTjc0o6aAPob-mCk7iaD03yZM4-_2sMGutGDnrF945s-fhROcuhqa0ENsf4Tt7vZyv3Ps3JAtekK8RB4aNdaKLE85INlMWQaEwfs/s1600/Living+room%252C+kitchen%252C+and+dining+room+furniture_tariff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="288" data-original-width="432" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVNQ5ny5XcXm6UhFqdRikie6uubu-wL9n416EU_4QnTjc0o6aAPob-mCk7iaD03yZM4-_2sMGutGDnrF945s-fhROcuhqa0ENsf4Tt7vZyv3Ps3JAtekK8RB4aNdaKLE85INlMWQaEwfs/s640/Living+room%252C+kitchen%252C+and+dining+room+furniture_tariff.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi21AtSWJs-soCIT5RC7wDskshrMzrld0i6S1s44g4nQgd7b1gQCTQPBpyzPGJP-f8FguhsQEn6TS0xIu8HU4ca4eICdRF_bFoIc9144n9_RwZAJHDALai66g8ANnplj8zV14U28663OHM/s1600/Other+furniture_tariff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="288" data-original-width="432" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi21AtSWJs-soCIT5RC7wDskshrMzrld0i6S1s44g4nQgd7b1gQCTQPBpyzPGJP-f8FguhsQEn6TS0xIu8HU4ca4eICdRF_bFoIc9144n9_RwZAJHDALai66g8ANnplj8zV14U28663OHM/s640/Other+furniture_tariff.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHUGWkDuCm657inYnqeaMvgel-uHX4km_urSrJK-USjhOrcYENFtWzs0M9pEZmtBg5nuKUYeuRS1Hggwd6B0muhuKlPzT-QWa0V8Vuith2tMiXqr3eH6CxdZ8eXXYNpE30SPIfnjfAJNI/s1600/Sewing+machines%252C+fabric+and+supplies_tariff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="288" data-original-width="432" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHUGWkDuCm657inYnqeaMvgel-uHX4km_urSrJK-USjhOrcYENFtWzs0M9pEZmtBg5nuKUYeuRS1Hggwd6B0muhuKlPzT-QWa0V8Vuith2tMiXqr3eH6CxdZ8eXXYNpE30SPIfnjfAJNI/s640/Sewing+machines%252C+fabric+and+supplies_tariff.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<h3 style="text-align: center;">
Tariff Unaffected Goods Counterfactuals (placebo)</h3>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNUhGvPi54-N3Bmwm5OmAl0I-e5zLTh7Wu6A7Hr7aoFv-o4CjjT_a3PdY9v3NmUtDMGCLP197zrBHQae3agTP2XrAc5MDe7Z3t_xMw44KPoR5-1ZDsMxsPMy-V1pB1pxrZJ5FDULlhayg/s1600/Alcoholic+beverages_non_tariff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="288" data-original-width="432" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNUhGvPi54-N3Bmwm5OmAl0I-e5zLTh7Wu6A7Hr7aoFv-o4CjjT_a3PdY9v3NmUtDMGCLP197zrBHQae3agTP2XrAc5MDe7Z3t_xMw44KPoR5-1ZDsMxsPMy-V1pB1pxrZJ5FDULlhayg/s640/Alcoholic+beverages_non_tariff.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5ueVqQGFuy_DtYLb529fHYCN6mctb17e1VcRDA5Z6R_Umxh3jrz-F2pjiS6O9A1Rdv8d05UuaMOpVXaHhtOlczbGwukOLyT1En98vjCKWu_-Kc1Ulkc9MToi4Z13FPM75_vz77w5Nzio/s1600/Cigarettes_non_tariff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="288" data-original-width="432" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5ueVqQGFuy_DtYLb529fHYCN6mctb17e1VcRDA5Z6R_Umxh3jrz-F2pjiS6O9A1Rdv8d05UuaMOpVXaHhtOlczbGwukOLyT1En98vjCKWu_-Kc1Ulkc9MToi4Z13FPM75_vz77w5Nzio/s640/Cigarettes_non_tariff.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwa2JEqq2Cw_ANq6fhWwPMQo7Ci6A3Uir9rQ-yg_EOTi9yurXSvHLErDkQm5Rn-Y8v1WIJMpXnsPh4X3pB8sMpgajpy9a_jT0O1OPYJz6LJ8MAXCiO26omjf8uYZY-bOP8QpwTpBLABI4/s1600/Citrus+fruits_non_tariff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="288" data-original-width="432" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwa2JEqq2Cw_ANq6fhWwPMQo7Ci6A3Uir9rQ-yg_EOTi9yurXSvHLErDkQm5Rn-Y8v1WIJMpXnsPh4X3pB8sMpgajpy9a_jT0O1OPYJz6LJ8MAXCiO26omjf8uYZY-bOP8QpwTpBLABI4/s640/Citrus+fruits_non_tariff.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9RK3VLLJ9UWA0yyGwOMLxdMKMCU5qVlUBBm16koD3K17jQHU0TjccHCeCPh9FWz8qnFnL66VFp9MuZE4o1DY6aLvJDZCo56LheNwaj2omlWSKMrAvydxoXGEyfb_ULh_Y_xUn47cpy60/s1600/Tires_non_tariff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="288" data-original-width="432" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9RK3VLLJ9UWA0yyGwOMLxdMKMCU5qVlUBBm16koD3K17jQHU0TjccHCeCPh9FWz8qnFnL66VFp9MuZE4o1DY6aLvJDZCo56LheNwaj2omlWSKMrAvydxoXGEyfb_ULh_Y_xUn47cpy60/s640/Tires_non_tariff.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRUEUba-6YKxS48vOootvyhmhC4_amdihLfI5ChkROZqACqHAupnTjrdwakyGrXiJOrcl6u0VQhvYvk9HnzahhvYZS6hMkpFtSNmXl-Ff_AGQzwEXqJLf21OZS_N0-wvGtNKw6-vYgsGg/s1600/Transportation+services_non_tariff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="288" data-original-width="432" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRUEUba-6YKxS48vOootvyhmhC4_amdihLfI5ChkROZqACqHAupnTjrdwakyGrXiJOrcl6u0VQhvYvk9HnzahhvYZS6hMkpFtSNmXl-Ff_AGQzwEXqJLf21OZS_N0-wvGtNKw6-vYgsGg/s640/Transportation+services_non_tariff.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8qyB6kFovD5wmv9DO9KMYL7osLAInzMz3vZK5gRVwsrYm-scwF2mM1ELlGvr26-yoZnsgb0wHWNd5wwGjjjc5D8xF9WADjTvI8q7fPOxbLBau0KrbGnuzN4QnApEypozNZjIYTL8eVCI/s1600/Women%2527s+apparel_non_tariff.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="288" data-original-width="432" height="426" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh8qyB6kFovD5wmv9DO9KMYL7osLAInzMz3vZK5gRVwsrYm-scwF2mM1ELlGvr26-yoZnsgb0wHWNd5wwGjjjc5D8xF9WADjTvI8q7fPOxbLBau0KrbGnuzN4QnApEypozNZjIYTL8eVCI/s640/Women%2527s+apparel_non_tariff.png" width="640" /></a></div>
<br />
<h2 style="text-align: center;">
</h2>
</div>
Vishal Misrahttp://www.blogger.com/profile/04748385949397825305noreply@blogger.com0tag:blogger.com,1999:blog-4160882474275537188.post-65519675343109816462019-05-05T09:53:00.000-07:002019-08-22T12:17:24.959-07:00mRSC: A new way to answer What Ifs and do time series prediction<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
<br /></div>
<h3 style="text-align: justify;">
Introduction</h3>
<div style="text-align: justify;">
<b><i>What if the federal minimum wage is raised to 16 dollars an hour?</i></b><br />
<i><b><br /></b>
<b>What if Steve Smith bats at number 5 in the Ashes 2019 instead of number 3?</b></i><br />
<i><b><br /></b>
<b>What if Australian style gun laws were implemented in the USA - what would be the impact on gun related violence?</b></i><br />
<i><b><br /></b>
<b>What if Eden Hazard attacks today instead of winging in the midfield?</b></i><br />
<div>
<br /></div>
<div>
<br /></div>
</div>
<div style="text-align: justify;">
"What if?” is one of the favorite questions that occupy minds, from sports fans to policymakers to philosophers. Invariably, there is no one answer to the What ifs and everyone remains convinced in their own alternate realities but a new wave of work has been looking at data-driven approaches to answer (at least a subset of) these What If questions. The mathematical tool of (Robust) Synthetic Control examines these What If questions by creating a synthetic version of reality and explore its evolution in time as a counterfactual to the actual reality.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Recently, together with my collaborators Jehangir Amjad (MIT/Google) Devavrat Shah (MIT) and Dennis Shen (MIT), we proposed a generalization of Robust Synthetic Control, the multi-dimensional Robust Synthetic Control (mRSC), and also demonstrated how the technique can be used to do time series predictions. The<a href="http://dna-pubs.cs.columbia.edu/citation/paperfile/233/mRSC.pdf" target="_blank"> paper </a>is appearing at the 2019 Joint <a href="https://www.sigmetrics.org/sigmetrics2019/" target="_blank">Sigmetrics/Performance conference</a>.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
As a fun application of the idea, we show how we can predict cricket scores accurately. We utilize the twin metrics of runs and wickets and cast it as an mRSC problem that enables us to predict the evolution of an innings with very little data. While we will continue to endlessly speculate What if Kumble had continued as the coach rather than Shastri, in this post we describe a mathematical tool that lets us predict the end of innings score after only a few overs!</div>
<div style="text-align: justify;">
<br /></div>
<h3 style="text-align: justify;">
</h3>
<h3 style="text-align: justify;">
Counterfactuals and Synthetic Control</h3>
<div>
<br /></div>
<div>
<br /></div>
<div style="text-align: justify;">
When evaluating the impact of a policy (e.g., gun control) on a metric of interest (e.g., crime-rate), it may not be possible or feasible to conduct a randomized control trial. In an ideal scenario, we would want two copies of reality, one with the policy and one without, and if there is a difference in the metric of interest in the two versions of reality we say that the policy had an impact. Given that we haven't conclusively shown the existence of multiverses, much less access them - what does one do? The problem statement is how does one estimate the <i>counterfactual</i>, i.e. simulate that other reality and measure that metric based only on observations of the <i>one</i> reality we have? A brilliant way to solve that problem was proposed by Abadie et al. called <a href="https://economics.mit.edu/files/11859" target="_blank">Synthetic Control</a>. The idea behind the technique is simple: assume that the change in policy happens in one clearly identified region (this change is called an intervention and the region is called the treatment unit) and several other regions exist which are correlated behaviorally to that region (but are not identical) which <i>do not</i> have that policy implemented and are called the donor pool (or <i>placebos</i>). An example is the policy of Proposition 99 implemented in 1988 in California, which levied a tax of 25 cents per pack of cigarettes. This additional tax not only made cigarettes more expensive to buy but the revenue raised by the tax was used to implement anti-tobacco programs throughout the state to raise awareness on the harms of tobacco and fund tobacco research. These twin effects were believed to have had a significant impact on cigarette consumption and the sales indeed went down, but how does one evaluate precisely if the tax had an impact or society had started frowning on smoking anyway because nationwide the sales were down? The insight was that California's behavior was indeed correlated to that of the other 49 states (more correlated with New York and Massachusetts than say Arizona or Texas), and one can construct a <i>synthetic</i> version of California using the other 49 states. More precisely, the time series of cigarette sales in California pre-Prop 99 was approximated as a linear combination of the sale of cigarettes in the other 49 states, where the weight of each state was determined by a regression process. The result of the analysis was striking and is shown in Figure 1: while cigarette sales did go down nationally and so also in the <i>synthetic</i> California, the real California had a much faster decrease in cigarette sales due to Proposition 99.</div>
<div style="text-align: justify;">
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCviupWb8JhB1NqzMhnipUftSjszboxCJGKp4NCzWHJku2Jx9u7DY5nT2aKscQrwtrWtmigtesYt5SSewN1WnV5lbdmRP51TEcRMeYle2gCmzftueKHr-mVv2W5E8pgteMwkxLRSoywGM/s1600/Screen+Shot+2019-05-05+at+7.23.48+AM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="692" data-original-width="954" height="290" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCviupWb8JhB1NqzMhnipUftSjszboxCJGKp4NCzWHJku2Jx9u7DY5nT2aKscQrwtrWtmigtesYt5SSewN1WnV5lbdmRP51TEcRMeYle2gCmzftueKHr-mVv2W5E8pgteMwkxLRSoywGM/s400/Screen+Shot+2019-05-05+at+7.23.48+AM.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1. Impact of Proposition 99 on Cigarette Sales in California</td></tr>
</tbody></table>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
<br /></div>
<h3 style="text-align: justify;">
Robust Synthetic Control (and Cricket)</h3>
<div style="text-align: justify;">
While the idea of Synthetic Control was brilliant and found a lot of success in estimating counterfactuals, it was also in some cases fragile and not robust to missing or noisy data. My collaborators at MIT developed a <a href="http://www.jmlr.org/papers/volume19/17-777/17-777.pdf" target="_blank">powerful latent variable model for Synthetic Control</a>, and made it robust against noisy and missing data. The key idea was that any observation is the sum of a function of two<i> latent </i>(i.e. hidden) <i>variables </i>and an independent noise term. The two latent variables are indexed by the donor pool, and time respectively. Then the entire dataset can be represented in a matrix where the rows correspond to the treatment unit and the donor pool units, and the y-axis corresponds to the time. The data-driven technique doesn't actually care what the variables are, or even what the nature of the function is, but is able to build an accurate model of the system based on the observations. That work formed the core of Jehangir's Ph.D. thesis. Jehangir is a cricket nut like me and his advisor Devavrat is my friend and we always wanted to work together but hadn't executed on that yet. Independently Devavrat had promised Jehangir that they would do at least <i>one </i>cricket related piece of work together. So Devavrat put 2 and 2 (or 11 and 11..) together, and called me one day with a proposal to work on a problem related to the biased behavior of the <a href="https://en.wikipedia.org/wiki/Duckworth%E2%80%93Lewis%E2%80%93Stern_method" target="_blank">Duckworth-Lewis-Stern </a>(DLS) method, a statistical system to account for the impact of rain interruptions in a cricket match. I had heard of the Synthetic Control method thanks to my wife who is a data scientist at FDNY but had no idea that Jehangir, Devavrat and Dennis had developed the Robust Synthetic Control method. So in my naïveté, I asked Devavrat "Have you heard of Synthetic Control? That might be the right tool for this problem". I don't know how hard Devavrat laughed when he saw my email, but here is his polite response to me:</div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEia5o1kvCRhhGeolUUki9u6zz0n3RH7Hwtut5HiKs6iSpcJyqvu6cSLKb_bhds1_bdNHtiuaWi6D5z-F26p3URYmnCF8tdnZD2ngqPRvzKVP37xjwMmnIBWBVXvkQ9K4ShUVk34CzSxTmo/s1600/Screen+Shot+2019-05-05+at+7.57.22+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="424" data-original-width="1600" height="168" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEia5o1kvCRhhGeolUUki9u6zz0n3RH7Hwtut5HiKs6iSpcJyqvu6cSLKb_bhds1_bdNHtiuaWi6D5z-F26p3URYmnCF8tdnZD2ngqPRvzKVP37xjwMmnIBWBVXvkQ9K4ShUVk34CzSxTmo/s640/Screen+Shot+2019-05-05+at+7.57.22+AM.png" width="640" /></a></div>
<div>
</div>
<div>
<br /></div>
<div>
<div style="text-align: justify;">
So we embarked on working together to build a model for cricket based on Robust Synthetic Control. Our metric of interest was the trajectory of runs scored by a team, with the "time" axis being the (sequential) ball of the game. The donor pool consisted of all the past ODI games that had happened thus far, and we devised a way to build a "synthetic" version of a game which had an intervention (rain-related interruption) based on the past games in the donor pool. Quickly we realized that looking only at the runs trajectory was not enough, we also needed to consider the wickets lost (a different metric, but is important for the evolution of the score metric) and initially we improved the model by filtering the donor pool to be of innings where the number of wickets lost was within +- one of the innings in question at the intervention point. However, we quickly converged to an even better way....</div>
</div>
<div>
<div style="text-align: justify;">
<br /></div>
</div>
<div>
<div style="text-align: justify;">
<br /></div>
</div>
<h3 style="text-align: left;">
mRSC: multidimensional Robust Synthetic Control </h3>
<div>
<br /></div>
<div style="text-align: justify;">
A fundamental issue (in our opinion) of the DLS system is that it is "Markovian", i.e. it only looks at what the current score is and doesn't account for <i>how</i> you reached the current score. So if you are 191/4, DLS would treat it the same if you were 22/3 and <i>recovered</i> to 191/4 or if you were 171/0 and <i>collapsed</i> to 191/4. Cricket, as with any other sport, is very momentum driven. The trajectory of the wicket losses has an important impact on the trajectory of the runs scored. So we realized that additional metrics are also useful in building the model of the system. Hence we changed our representation of the observation as a function of three (latent) variables - one corresponding to the treatment or donor unit, one corresponding to the "time", and finally one corresponding to the measured metric; plus the independent noise term. The model then moved from a matrix to a third-order tensor. We showed that Robust Synthetic Control then becomes a special case of mRSC (when the number of metrics considered is 1). The details are in the paper and are not crucial to understand the concept so I won't present them here. The critical point is that by fusing together multiple related metrics, we are able to build a much more accurate representation of the system, requiring a much shorter dataset pre-intervention.</div>
<div style="text-align: justify;">
<br /></div>
<h3 style="text-align: left;">
Validating the technique and time series prediction</h3>
<div style="text-align: justify;">
Once we have the formulation and built the model, a natural question is how does one <i>validate</i> the model? Recall that the original application was to estimate counterfactuals and for that one needs to not only identify specific interventions but know the exact impact of them and compare that against the multi-dimensional synthetic model that we built. Although it is possible to use the pre-intervention data to cross-validate the performance of any estimation method, such a methodology ignores the period of interest: the post-intervention period. An alternate and more effective approach is to study the performance of an estimation method on units that do not experience the intervention, i.e., the <i>placebo</i> units. If the method is able to accurately estimate the observed post-intervention evolution of the placebo unit(s), it would be reasonable to assume that it would perform well in estimating the unobserved counterfactuals for the unit of interest. So we performed extensive validation of the technique by comparing the evolution of metrics of interest for a variety of "placebo" units in various settings, and in the process developed a <b>general purpose time series prediction</b> algorithm. As long as the notion of time is <i>relative</i>, i,e, the donor pool has already undergone the future evolution in "time", our synthetic model will be able to predict the future evolution of the unit of interest. Examples of such scenarios are sporting contests, where the "time" is say balls of a game or minute of a contest and the donor pool consists of historical games, or the financial projections of revenues of a company where the "time" axis is month of the year or quarter, and donor pool consists of the corresponding periods in prior years. It can also be used to predict things like traffic on a site, where you have daily data for prior years (including spikes generated by periodic events like holidays etc.). One could predict the traffic generated in a datacenter by the regular launch of computational tasks like map reduce jobs etc. In short, any scenario where there is some repeating (but not exact) pattern, our technique can build a model of the current system by incorporating information of past instances of the same pattern. Some examples of our scheme in action are below, you can find more details in the paper:</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhH3BgBRmq9zGamUhd3CzEaK_fhitmjUe3IHU8zHet7pzzDj1p4f5bjITHyPf0uDan6smCcrmMubIl3mvAQXvKb9ij4Ynr8XeTVXnjpMwhjhZjv_18xtoENeTLyXjy-zhqSirzFOLmBkg0/s1600/store1_dept23_15_mrsc.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="278" data-original-width="405" height="273" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhH3BgBRmq9zGamUhd3CzEaK_fhitmjUe3IHU8zHet7pzzDj1p4f5bjITHyPf0uDan6smCcrmMubIl3mvAQXvKb9ij4Ynr8XeTVXnjpMwhjhZjv_18xtoENeTLyXjy-zhqSirzFOLmBkg0/s400/store1_dept23_15_mrsc.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
Figure 2: Predicting sales in a department of a Walmart store. Multiple metrics are sales in different departments, donor pool is data from other stores. Red line is time at which prediction is made</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjxv3BezBL_q9ifcBN-uTjBGrlLBAn77LxuJx3B9X92fXubO6jHZCUz73hPnwZIuMu4e7FD6Nh5Xm6ON9qm1qkHiYbAEAX2VSEIEVbww_UzpXgB7GCBBvzh4QtLUJg0PY1bxtvEihUvIk/s1600/zim-aus-2-30.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="480" data-original-width="640" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjxv3BezBL_q9ifcBN-uTjBGrlLBAn77LxuJx3B9X92fXubO6jHZCUz73hPnwZIuMu4e7FD6Nh5Xm6ON9qm1qkHiYbAEAX2VSEIEVbww_UzpXgB7GCBBvzh4QtLUJg0PY1bxtvEihUvIk/s400/zim-aus-2-30.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
Figure 4: Final score of the innings in an Aus-Zim ODI in 2004. Multiple metrics are wickets and runs, the donor pool is historical ODIs. Prediction made at 30th over.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwhodpvDRsmEfg6v_TZLfLOr3Pzs13jIcJpZGbQLvTqYbdhe-UJLwaNFAxryVaqkFhP1F7cvL9_ZoFinCwlFEdxurBjR7pQVJdWcj8SwMxpe5s4KT9DFnl7m3ZpN_3u3EusN9EznShjW4/s1600/1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="278" data-original-width="396" height="280" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwhodpvDRsmEfg6v_TZLfLOr3Pzs13jIcJpZGbQLvTqYbdhe-UJLwaNFAxryVaqkFhP1F7cvL9_ZoFinCwlFEdxurBjR7pQVJdWcj8SwMxpe5s4KT9DFnl7m3ZpN_3u3EusN9EznShjW4/s400/1.png" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
Figure 5: Final score of a team in an NBA game. Multiple metrics are the score trajectories of team 1 and score trajectories of team 2 (other team). Donor pool is past NBA games. Score updated ever 15 seconds, prediction made at the end of 1st quarter.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<h3 style="clear: both; text-align: justify;">
Code and examples</h3>
<div>
As I have tried to explain, our technique is a very general purpose one applicable to a variety of scenarios for both counterfactual estimations as well as time series prediction. If this interests you and you want to try it out, Jehangir has made his code available on <a href="https://github.com/jehangiramjad/tslib" target="_blank">github</a>. Please feel free to download and play around with it. I began the post with a few counterfactuals that we can imagine estimating with the technique, and I'll bookend it with some example of time series predictions one can do:<br />
<br />
<b><i>I have sales data for my company for the past several years. Based on the first three months of sales, what do the projections for the rest of the year look like?</i></b><br />
<i><b><br /></b>
<b>CSK is 76/3 at the end of 9 overs. What is going to be their end of innings score?</b></i><br />
<i><b><br /></b>
<b>The traffic on my network is growing. I have data from the first few months of the year - what kind of capacity planning I need to do to accommodate growth by the end of the year?</b></i><br />
<i><b><br /></b>
<b>Golden State is leading against the Bucks 28-26 at the end of the 1st quarter. Who is going to win and what would be the score?</b></i></div>
<div>
<b> </b></div>
</div>
Vishal Misrahttp://www.blogger.com/profile/04748385949397825305noreply@blogger.com24tag:blogger.com,1999:blog-4160882474275537188.post-40584295758677243522015-12-29T14:33:00.001-08:002015-12-31T00:55:31.489-08:00Half the equation and half the definition<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">There is a lot of confusion over what constitutes Net Neutrality, so much so that parties fiercely on the opposite side of issues both claim to be for it. As an example, the current controversy over Free Basics has been between Facebook, whose CEO penned an Op-Ed entitled "<a href="http://blogs.timesofindia.indiatimes.com/toi-edit-page/free-basics-protects-net-neutrality/" target="_blank">Free Basics protects net neutrality</a>", and on the opposite side of it is a volunteer coalition, SaveTheInternet (STI), whose entire charter is to protect Net Neutrality. As the Op-Ed from the volunteers <a href="http://blogs.timesofindia.indiatimes.com/toi-edit-page/its-a-battle-for-internet-freedom/" target="_blank">suggests</a>, the basic contention between Facebook and the volunteers is a different definition of Net Neutrality. While the concept of Net Neutrality was coined by Tim Wu back in 2003, the definition of what constitutes Net Neutrality has been evolving. </span></div>
<br />
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">Let me walk you through the evolution of the definition that the STI coalition is going with, which is widely accepted and which I have arrived at after years of researching the issue. I will explain why Facebook (amongst countless others, they are not solely to blame here) only consider half the equation and thus end up with half the definition.</span></div>
<br />
<br />
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">I'll start with the folk-definition that we started hearing, around 10 years ago:</span></div>
<br />
<h4 style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></h4>
<h4 style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">Folk Definition: All packets must be treated equally</span></h4>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">As networking researchers we knew that this definition was not practical and it made little sense to us. Without getting too much into boring details, we knew routers on the Internet did not treat all packets identically (TCP-SYN packets are treated differently from TCP-Data or TCP-Ack packets, UDP packets are treated differently, packets in the tail of the queue are dropped during congestion etc. etc.). However, we also knew what the principle of Net Neutrality was trying to say, and that was the network did not discriminate. So the folk definition needed to be made crisper. The FCC adopted Net Neutrality rules last year and the definition broadly laid out the following principles:</span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
</div>
<h4 style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">FCC: ISPs will not block or throttle any traffic and will not implement any paid prioritization (no fast lanes)</span></h4>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">This definition changes the abstraction from how packets are treated, to how services are treated which is a logical progression. However the FCC missed out in one crucial aspect, and that is not incorporating the concept of differential pricing in its Net Neutrality Principles. Zero Rating, which is a special case of differential pricing, was not a big problem in the US when the Open Internet order was voted upon, and the FCC preferred a wait and watch approach to it (as a refresher, Zero Rating is the concept where consumers don't pay for the bandwidth of some or all services, and instead the cost of the bandwidth is borne either by the ISP or the content provider). The FCC definition focused on <i>quality of service</i> (QoS) as the determining factor for Net Neutrality, and insisted that all content on the Internet received the same quality of service from ISPs. The intent was to not provide competitive advantage to any service on the Internet, as that was in the best interest of both consumers as well as entrepreneurs. However it missed out in the following way:</span></div>
<br />
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">As a brief background, in game theory (the mathematical tool we have used in our work on analyzing the issue), the quantity that we focus upon is called <i><a href="https://en.wikipedia.org/wiki/Economic_surplus" target="_blank">Consumer Surplus</a>. </i>Surplus is defined as Utility derived from a particular service </span><i style="font-family: Arial, Helvetica, sans-serif;">minus</i><span style="font-family: Arial, Helvetica, sans-serif;"> the </span><b style="font-family: Arial, Helvetica, sans-serif;">cost paid</b><span style="font-family: Arial, Helvetica, sans-serif;"> to obtain that service. The Utility is a mathematical quantity that models the impact of the QoS obtained for a particular application, and the FCC was absolutely correct in enforcing neutrality there, but the FCC did not model the </span><i style="font-family: Arial, Helvetica, sans-serif;">cost paid</i><span style="font-family: Arial, Helvetica, sans-serif;"> in its definition of Net Neutrality </span><span style="font-family: Arial, Helvetica, sans-serif;">(and it is the definition Facebook uses to justify Free Basics as being consistent with Net Neutrality)</span><span style="font-family: Arial, Helvetica, sans-serif;">. How much an application costs changes the surplus a consumer obtains, and applications with similar utility (quality) but with differing costs provide different surpluses. In game theoretic models higher surpluses get competitive advantages, thus it is </span><i style="font-family: Arial, Helvetica, sans-serif;">crucial</i><span style="font-family: Arial, Helvetica, sans-serif;"> to model the cost aspect of an application to get to a definition of Net Neutrality that works. Differential pricing or Zero Rating of select services absolutely violates the principle of Net Neutrality if we consider the impact on consumer surplus.</span></div>
<br />
<br />
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">Thus, if we only model half the equation, we end up with a definition of Net Neutrality that focuses </span><i style="font-family: Arial, Helvetica, sans-serif;">only</i><span style="font-family: Arial, Helvetica, sans-serif;"> on QoS, however if we model the equation fully then the price of the service comes into play. A lot of people only model half the equation (Facebook included) and thus claim that differential pricing (Zero Rating specifically) is fine under Network Neutrality, but that is not true. If we are talking about a true level playing field, then the other half of the equation cannot be ignored.</span></div>
<br />
<br />
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">Access Now, a global non-profit aimed at protecting the digital rights of citizens, has adopted a definition that states the following:</span></div>
<br />
<h4 style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></h4>
<h4 style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">Access Now: </span><span style="font-family: Arial, Helvetica, sans-serif;">Net neutrality requires that the Internet be maintained as an open platform, on which network providers treat all content, applications and services equally, without discrimination.</span></h4>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">This definition implies that differential pricing cannot be adopted, but it does not say so explicitly and people (usually differential pricing advocates) can easily ignore the pricing aspect of a service and say Zero Rating is consistent with this definition. To fix this minor issue, and make things explicit, I have proposed the following definition which has received acceptance from academics, policy makers, entrepreneurs and activists alike, and I announced it publicly sometime back:</span></div>
<blockquote class="twitter-tweet" lang="en">
<div dir="ltr" lang="en">
<div style="text-align: justify;">
Internet is a platform where ISPs provide no competitive advantage to specific apps/services, either through pricing or QoS
<a href="https://twitter.com/hashtag/NetNeutrality?src=hash">#NetNeutrality</a></div>
</div>
<div style="text-align: justify;">
— Vishal Misra (@vishalmisra) <a href="https://twitter.com/vishalmisra/status/631285727024672768">August 12, 2015</a></div>
</blockquote>
<br />
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">This definition has the following properties</span></div>
<script async="" charset="utf-8" src="//platform.twitter.com/widgets.js"></script>
<br />
<div style="text-align: justify;">
<br /></div>
<ol style="text-align: left;">
<li style="text-align: justify;"><span style="font-family: Arial, Helvetica, sans-serif;">It incorporates both QoS and Pricing in the definition of Net Neutrality, thus correctly modeling consumer surplus.</span></li>
<li style="text-align: justify;"><span style="font-family: Arial, Helvetica, sans-serif;">It makes explicit the notion that Net Neutrality is not how we treat packets but how we treat competition.</span></li>
<li style="text-align: justify;"><span style="font-family: Arial, Helvetica, sans-serif;">It allows for reasonable traffic management by ISPs without violating Net Neutrality.</span></li>
<li style="text-align: justify;"><span style="font-family: Arial, Helvetica, sans-serif;">It allows <i>differential</i> QoS and/or pricing as long as it is allowed in a non-discriminatory way. ISPs can prioritize <i>all</i> real time traffic (e.g. all voice or all Video Conference traffic in a provider agnostic way) over all non-real time traffic. Similarly all emergency services or health monitoring apps can be prioritized.</span></li>
<li style="text-align: justify;"><span style="font-family: Arial, Helvetica, sans-serif;">It allows creating and differentially pricing entire class of services. For instance, an ISP can create an extremely low latency service and offer it to all games/gamers without discrimination, and that should be fine.The definition permits differentiation between services, but prohibits discrimination within a service.</span></li>
<li style="text-align: justify;"><span style="font-family: Arial, Helvetica, sans-serif;">It ensures a level playing field on the Internet, where upstarts can come in and compete on the basis of ideas.</span></li>
<li style="text-align: justify;"><span style="font-family: Arial, Helvetica, sans-serif;">Lastly, and this is only half in jest, the definition fits in 140 characters including a hashtag.</span></li>
</ol>
<div style="text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif;">I am a big believer in the power of capitalism, as I think humans are largely selfish with varying degrees of altruism. However, for capitalism to work for the greater good of society, it is critical that the selfish interests of corporations align with public interests. And that's where regulators step in, using the concept of <a href="https://en.wikipedia.org/wiki/Mechanism_design" target="_blank">mechanism design</a>, to introduce a minimal set of regulations that incentivize corporations to act in societal interest. I think the concept of Net Neutrality, defined in the way above, provides the mechanism for the Internet economy to work in the public interest. I hope the right regulations get passed.</span></div>
<div style="text-align: justify;">
<br /></div>
<br /></div>
Vishal Misrahttp://www.blogger.com/profile/04748385949397825305noreply@blogger.com1tag:blogger.com,1999:blog-4160882474275537188.post-64087830277467527082015-12-26T03:29:00.003-08:002015-12-26T03:29:56.255-08:00The business of ZeroRating<div dir="ltr" style="text-align: left;" trbidi="on">
ZeroRating conversations are dominating Network Neutrality issues these days, whether it is the <a href="http://bits.blogs.nytimes.com/2015/12/23/indian-regulators-suspend-facebooks-free-basic-services/?_r=0" target="_blank">FreeBasics controversy</a> in India, <a href="http://bits.blogs.nytimes.com/2015/12/17/f-c-c-asks-comcast-att-and-t-mobile-about-zero-rating-services/" target="_blank">Binge On</a> by T-Mobile, or <a href="http://arstechnica.com/business/2015/12/verizon-to-test-sponsored-data-let-companies-pay-to-bypass-data-caps/" target="_blank">Verizon's recent announcement</a> of a plan similar to AT&T's sponsored data. Here are a few thoughts to consider about ZeroRating and why it makes no sense (to me).<br />
<br />
If ISPs Zero Rate content, somebody has to pay for the bandwidth. Suppose the Content provider pays for it. Then there is a pricing problem:<br />
<br />
<ul style="text-align: left;">
<li>ISPs <b>cannot</b> charge the content provider a price <b>above </b>the price they charge consumers. Suppose they charge consumers X per MB of data, and they charge content providers X+Y per MB of data. Then, for sufficient traffic where overheads are accounted for, it is <i>cheaper</i> for content providers to send recharge coupons back directly to the customers who used their services. Long term, pricing above the consumer price is not sustainable.</li>
<li>ISPs <b>cannot</b> charge the content provider a price <b>below</b> the price they charge consumers. Suppose they charge consumers X per MB of data, and they charge content providers X-Y per MB of data. Then if the plan is truly open, a company like <a href="http://www.gigato.co/" target="_blank">Gigato</a> can come along, buy data in volume and become a virtual ISP. They can funnel traffic to services via their servers (they can remain good guys and not decrypt or store private data), sell the bandwidth to consumers at X-Y/2 and pocket the difference. The ISPs lose out.</li>
</ul>
Or alternately, the ISP pays for the bandwidth of the content.<br /><ul style="text-align: left;">
<li>This opens the possibility of <a href="http://news.investors.com/122215-786473-verizon-plans-premium-mobile-video-skirting-net-neutrality-rules.htm" target="_blank">vertical integration</a>, where ISPs ZeroRate their own content, and that is <b>extremely</b> bad for competition. Or ISPs ZeroRate only a select group of content providers, for non-transparent reasons (FreeBasics or Binge On "technical" requirements that make the walled gardens implicitly closed), leading to a fractured experience/Internet for their consumers.</li>
</ul>
<div>
<br /></div>
It is not clear to me what the business model is for ZeroRating, where the ISPs make money and provide an Open and Neutral Internet experience for their consumers. Economic issues are really the core of Network Neutrality, and nobody has explained to me how the economic model of ZeroRating remains consistent with Network Neutrality<br /><br />
<h4 style="text-align: left;">
<br /></h4>
</div>
Vishal Misrahttp://www.blogger.com/profile/04748385949397825305noreply@blogger.com87tag:blogger.com,1999:blog-4160882474275537188.post-38304470443945243522015-02-06T08:02:00.002-08:002015-02-06T08:10:00.608-08:00The most important aspect of the new FCC proposal on Net Neutrality<div dir="ltr" style="text-align: left;" trbidi="on">
So it has happened. FCC Chairman Wheeler has announced that he wants to bring broadband (both wired and wireless) under Title-II provisions. Network Neutrality advocates are <a href="http://www.nytimes.com/roomfordebate/2015/02/04/regulate-internet-providers/the-internet-is-back-to-solid-regulatory-ground" target="_blank">ecstatic</a>, hailing him <a href="http://www.newyorker.com/news/news-desk/net-neutrality-shows-democracy-can-work" target="_blank">as a hero</a>. The fact sheet outlining the FCC proposals is available <a href="http://transition.fcc.gov/Daily_Releases/Daily_Business/2015/db0204/DOC-331869A1.pdf" target="_blank">here</a>.<br />
<br />
I found myself initially disappointed, and thought that this might actually be a retrograde step for consumers. The reason for that was the <b>Bright Line Rules</b> that the FCC highlighted: no blocking, no throttling and no paid prioritization were not actual, real problems. ISPs don't block (that's the job of governments), ISPs do not explicitly throttle - there are sophisticated ways of achieving <a href="http://www.newyorker.com/news/news-desk/net-neutrality-shows-democracy-can-work" target="_blank">the same result</a> and I don't know of any actual "paid prioritization" (i.e. QoS provided for specific content on routers/switches in exchange for money). The real problem is paid peering which does not come under the purview of the Bright Line Rules.<br />
<br />
Additionally, the forbearance that has been explicitly stated includes "no unbundling" which is not pro consumer. The proposal also completely ignores "<a href="http://en.wikipedia.org/wiki/Zero-rating" target="_blank">zero rating</a>" which is the functional equivalent of fast lanes and could be a big problem down the road.<br />
<br />
However, something that I missed but went back to when I was reading <a href="http://arstechnica.com/business/2015/02/dont-call-them-utility-rules-the-fccs-net-neutrality-regime-explained/" target="_blank">a post by Jon Brodkin</a>, is the 4th bullet point of the provisions of Title-II that <b>will</b> be enforced. It says:<br />
<br />
<br />
<div class="page" title="Page 3">
<div class="layoutArea">
<div class="column">
<blockquote class="tr_bq">
<span style="font-family: CourierNew; font-size: 11pt;">o </span><span style="font-family: TimesNewRoman; font-size: 11pt;">Ensures fair access to poles and conduits under Section 224, which would boost the </span><span style="font-family: TimesNewRoman; font-size: 11pt;">deployment of new broadband networks </span></blockquote>
<span style="font-family: TimesNewRoman; font-size: 11pt;"><br /></span>
This is the single most important part of the new FCC proposal. My initial reaction has been that FCC has done nothing to promote competition, but I take it back. We have always maintained that <a href="http://peerunreviewed.blogspot.com/2014/02/why-is-netflix-paying-comcast.html" target="_blank">competition is the real issue</a>, not network neutrality and there are many others who say the same thing, e.g. from Jon's publication Ars Technica there is an excellent post by Peter Bright saying <i><a href="http://arstechnica.com/tech-policy/2014/06/we-dont-need-net-neutrality-we-need-competition/" target="_blank">We don’t need net neutrality; we need competition</a>. </i>Fair access to Poles and Conduits is critical to increasing broadband competition in the US, and that's the real problem to be solved, not enforcement of "Net Neutrality".</div>
</div>
</div>
<br />
<div>
<div class="page" title="Page 2">
<div class="layoutArea">
<div class="column">
<ul style="list-style-type: none;">
<li><br /></li>
</ul>
</div>
</div>
</div>
</div>
</div>
Vishal Misrahttp://www.blogger.com/profile/04748385949397825305noreply@blogger.com2tag:blogger.com,1999:blog-4160882474275537188.post-26047980461548851832015-02-05T09:03:00.000-08:002015-02-06T07:24:42.364-08:00My appearance on BBC World News discussing Net Neutrality<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="text-align: left;">
I appeared on the BBC World News program Global with Matthew Amroliwala on February 5th, 2015 and spoke about Net Neutrality and the new FCC proposals. I tried to make the point that Net Neutrality is a symptom and the real issue is (lack of) competition.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The clip below is courtesy the BBC.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe width="320" height="266" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/tsGlTCgQ9cA/0.jpg" src="http://www.youtube.com/embed/tsGlTCgQ9cA?feature=player_embedded" frameborder="0" allowfullscreen></iframe></div>
<br /></div>
<div style="text-align: left;">
<br /></div>
<br />
<br /></div>
Vishal Misrahttp://www.blogger.com/profile/04748385949397825305noreply@blogger.com2tag:blogger.com,1999:blog-4160882474275537188.post-43181610383400186612014-10-17T08:02:00.000-07:002014-10-26T07:13:46.497-07:00We need an "AppleSim" for Wired Internet<div dir="ltr" style="text-align: left;" trbidi="on">
Our work has long argued that <i><a href="http://www.cs.columbia.edu/2014/net-neutrality/" target="_blank">competition</a></i> is the key to an Open Internet, not Network Neutrality regulations. The introduction of AppleSim by Apple on their iPads is a very interesting and important development in that regard. As Apple says in their <a href="http://www.apple.com/ipad-air-2/wireless/" target="_blank">marketing literature</a>:<br />
<br />
<blockquote class="tr_bq">
<span style="background-color: #f9f9f9; color: #8e8e8e; font-family: 'PT Serif', Georgia, serif; line-height: 30px;">So whenever you need it, you can choose the plan that works best for you—with no long-term commitments</span></blockquote>
Apple's market power (ironically) has ensured that they got 3 of the 4 top wireless carriers in the US to sign on as partners. If you are not happy with the service of a provider, you can <i>easily</i> switch.<br />
<br />
Now imagine if a similar situation existed for wired Internet access for consumers. Everyone had the equivalent of an "AppleSim" at home. Say in the form of a router, that can easily connect to different wired Internet providers with the click of a button on a web page. You are not happy with Netflix speed by your provider FIOS? Fine - you switch to Comcast. Your ISP blocks Skype video calls? Fine - switch to an ISP that doesn't and is eager to get your business. It would mean the end of <a href="http://peerunreviewed.blogspot.com/2014/02/why-is-netflix-paying-comcast.html" target="_blank">peering disputes</a> like the one we have seen in the past that are a result of the market power of last mile monopolies. ISPs would compete with each other to get your business based on the services they offer and the quality of their connection. Right now, the ISPs seem to be competing with each other in exploiting their last mile monopolies to extract revenue from content providers.<br />
<br />
Laying down the infrastructure to make this vision of an "AppleSim" for wired access is something complex and involved and for another day. However the equivalent of a company like Apple with the "market power" to flex their muscle and get ISPs to fall in line are municipalities. <a href="http://peerunreviewed.blogspot.com/2014/02/the-public-option-municipal-broadband.html" target="_blank">Municipal Internet</a> is one way to ensure that there is enough competition at the last mile for consumers. Another closely related approach is what Stockholm has done with <a href="http://www.stokab.se/In-english/" target="_blank">stokab</a>, they have created a shared last mile infrastructure that different providers can lease and get to customers. We need to get a similar concept going in the US, on a large scale. <i>That</i> would ensure an Open Internet, not difficult to define and impossible to regulate ideals like "Network Neutrality".<br />
<br />
<h4 style="text-align: left;">
Update:</h4>
<div>
AT&T has announced that when customers activate their AppleSim using the AT&T network they will <i><a href="http://techcrunch.com/2014/10/24/att-breaks-the-apple-sims-best-feature-locks-it-down-if-you-pick-att/" target="_blank">lock</a></i> the SIM. There goes easy switching to different providers. A retrograde step by AT&T and Apple should push back.</div>
</div>
Vishal Misrahttp://www.blogger.com/profile/04748385949397825305noreply@blogger.com3tag:blogger.com,1999:blog-4160882474275537188.post-55270327282265694382014-07-13T17:09:00.000-07:002014-07-14T06:50:01.764-07:00Our response to FCC's NPRM on Open Internet (14-28)<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="MsoNormal">
The following is the detailed response we have submitted to the FCC on it's NPRM <a href="http://www.fcc.gov/document/protecting-and-promoting-open-internet-nprm" target="_blank">14-28 Protecting and Promoting the Open Internet</a> (aka Net Neutrality proposals).</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br />
Response to the FCC NPRM on Open Internet prepared by
Richard Ma (<a href="mailto:tbma@comp.nus.edu.sg">tbma@comp.nus.edu.sg</a>) and
Vishal Misra (<a href="mailto:misra@cs.columbia.edu">misra@cs.columbia.edu</a>)
<o:p></o:p></div>
<div class="MsoListParagraphCxSpFirst" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">1.<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->Para. 1: From the first sentence, the proposed
rulemaking emphasizes “broadband investment and deployment”. It seems that the
scope is really limited to broadband or eyeball ISPs. We want to emphasize that
the net neutrality issues are not limited to broadband. Many core issues are
involved with transit ISP and content providers as well. When it mentions
“open”, does it mean “neutral”? If it is not “open”, is the current Internet
“closed”? It seems that “open” is not well-defined. We comment that “open”
should be defined more clearly.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">2.<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->Para. 2: “What is the right public policy to
ensure that the Internet remains open?” Again, to answer this question, we
first need to be clear about the definition of “open”. <o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">3.<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->Para. 3: The no-blocking rule is too limited. An
ISP could give extremely small amount of capacity for certain content traffic,
which does not block it; however, the resulting quality might be too poor for
users to make use of the content. “Commercially unreasonable actions” needs a
clear definition and interpretations. <o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">4.<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->Para. 4: “the best ways to define, prevent … the
practices that threaten an open Internet”. We suggest that the openness should
be defined based on the level of market competition. Our work [1] has shown
that whether regulations are needed really depends on whether market competition
exists or not. <o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">5.<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->Para. 6: In general, access ISPs do have
incentives to maximize their revenue by differentiating services, which might
or might not be beneficial for the end-users; however, this is different from
being “open” or not. There is a zero-sum game between the edge and access
providers, therefore, it ultimately is a profit sharing problem [3, 4]. With
respect to the second point, our recent work [2] studies paid prioritization on
edge providers and from a social welfare perspective, our results support the
use of priority-based pricing and service differentiation rather than imposing
net neutrality regulations. With respect to the third point, our work [1]
confirms that without market competition, a monopoly ISP does have incentives
to make free ordinary services “damaged good”; however, market competition
could avoid such a scenario.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">6.<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->Para. 10: “First, we generally propose to retain
the definitions and scope of the 2010 rules.” We comment that the scope could
be broadened and the definition of “open” should be made more clearly. “Second,
… should enhance the transparency rule …” We agree with this which will enhance
the competition among the ISPs. “Third, … no-blocking rule … with a revised
rationale, in order to ensure that all end users and edge providers can enjoy
the use of robust, fast and dynamic Internet access.” It seems that the
proposed requirement is a lot more than a no-blocking requirement. “Fourth, …
to adhere to an enforceable legal standard of commercially reasonable practices
…, asking how harm can best be identified and prohibited and whether certain
practices, like paid prioritization, should be barred altogether.” Again, we
comment that a more clear definition of “commercially reasonable practice” is
needed. In our work [1], we show that paid prioritization does not harm user
welfare unless the access ISP is a monopoly which makes a lower-class service
“damaged good”. We suggest that a service comparison between the different
service classes can be used as a guideline to limit service differentiations.
In our recent work [5], we propose to limit the quality difference, rather than
the absolute quality level, to ensure the “commercial reasonable actions”. “Fifth,
… dispute resolution process”. Our prior work [3, 4] looked into the ISP
settlement issues and identify that the Shapley mechanism could be an ideal
mechanism to arbitrate disputes among interconnecting ISPs to maximize the
social welfare. This can be practically difficult so the correct approach
should be to increase competition at the access which obviates the need for any
dispute resolution.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">7.<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->Para. 12: “consumers’ hunger for high-value
content”. Net neutrality could hurt when low-value content compete with
high-value content and reduce them via negative network effect, i.e.,
congestion. Allowing ISPs’ differentiation could prioritize high-value content
to end-users.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">8.<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->Para 13: the original purpose is to “encourage
broadband deployment”. Too restrictive a policy will reduce the incentives of
broadband providers to deploy in suburban areas.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">9.<span style="font: 7.0pt "Times New Roman";">
</span></span></span><!--[endif]-->Para 17: “stop blocking” is easy to achieve,
however, the real question is about service differentiation. Could we allow
ISPs to limit the throughput of certain applications? Maybe we could not let
ISPs to “actively” limit the throughput, but they could maintain different
service classes and let edge providers to choose the services. We proposed and
studied this type of “passive” service differentiation in our work [1, 2, 5],
and our results in general support ISPs to provide such differentiations as
long as enough market competition exists.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">10.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para
18: it all boils down to the question whether the network management is
reasonable or not. By measuring the relative quality (throughput) between BT
traffic and other traffic, as proposed in [5], one could infer whether the
network management is reasonable. If we only look at the quality of BT traffic,
even it gets low throughput, it might be because the system is limited in
capacity and cannot support the demand (not because of ISP’s differentiation).<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">11.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
20: “nondiscrimination and transparency rules” We suggest that
nondiscrimination should be defined to allow passive service differentiation
[1, 2, 5], e.g., class-based service differentiation, under which content
providers get to choose what service to use.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">12.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
21: we want to comment that the 2<sup>nd</sup> case of “blocking” is a special
case of the 3<sup>rd</sup> scenario of “unreasonable discriminating in
transmitting lawful network traffic”. We need a definition of “unreasonable
discrimination” and suggest use the quality difference in different services.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">13.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
23: “… D.C. Circuit … grant the Commission affirmative authority to encourage
and accelerate the deployment of broadband capability to all Americans through,
among other things, measures that promote competition in the local
telecommunications market or remove barriers to infrastructure investment.” We
totally agree with this view. Based on the results of our work [1, 2], we
suggest that the FCC should promote competition rather than imposing
restrictive rules on ISPs. “the court struck down the anti-blocking and
anti-discrimination rules, explaining that the Commission had chosen an
impermissible mechanism … imposed per se common carriage requirements … ” Based
on our work [5], we suggest that the FCC should not impose strict service
“requirements” on the providers. However, they could impose rules to restrict
the quality difference among service classes. On the one hand, through this
type of regulation, it does not impose a common carriage requirement. On the
other hand, this type of regulation is also enforceable and provides ISPs
spaces to differentiate services. <o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">14.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
26: we want to comment that certain passive service differentiation does not
necessarily mean “restricting edge providers’ ability to reach users …”. Our
prior work [1] showed that the “short-term incentive to limit openness” does
exist under a monopolistic market. However, if market competition is enough,
this behavior will reduce the market share of the ISPs.<span style="mso-spacerun: yes;"> </span>We also showed that under market competition,
ISPs will have incentives to differentiate services, resulting in an increase
in the social welfare. <o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">15.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
27: “… any small costs of imposing the rules were outweighed by the positive
effect on network investment from the preservation of the openness that drives
the virtuous circle …”. We have mixed feelings on this. A strict rule might
reduce the investment incentives of access ISP to deploy capacity to rural
areas. <o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">16.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
32: “tremendous growth in the online video markets … revenues from online video
services grew 175% from $1.86 to $5.12 billion …” This causes congestion and
unbalance in profit sharing, which reduces the investment incentives from
access ISPs. Our prior work [3, 4] on Shapley profit-sharing indicates that a
re-balancing is needed to compensate the access ISPs from the content side
ISPs. This can be achieved in two ways, either by paid-peering arrangements or
by increasing competition at the access.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">17.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
33: we agree that competition should be encouraged; however, without incentives
for ISPs to do service differentiation, they might not have any incentives to
deploy infrastructure to rural areas. Also, natural monopoly exists in those
areas, because a 2<sup>nd</sup> provider is basically too expensive. Regulating
this natural monopoly could be different from regulating other ISPs; otherwise,
no ISPs will be willing to deploy infrastructures in rural areas.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">18.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]--><span style="mso-spacerun: yes;"> </span>Para. 34: we want to comment on the role of
the Internet’s openness in facilitating innovation, economic growth,
competition and broadband investment and deployment. Here, we have conflicting
roles of competition and broadband investment. Increasing competition on the
broadband side might reduce their incentives to invest, but encourage more
investments on the edge/content providers. However, lack of investment in
infrastructure will eventually hurt the whole ecosystem, innovation and
economic growth. We need to balance the different objectives. One solution is
to look at both broadband providers and edge providers as a whole Internet
supply chain, and think about the revenue/profit sharing among them, so as to
maximize the utility of the overall ecosystem. Under such a macro model, we
might allow broadband providers to differentiate services and charge edge
providers. <o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">19.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
36: the scope of the market between broadband and end-users is limited. No
evidence exists for “pay-for-priority” on end-users. However, broadband is
doing that on the peering side.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">20.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
37: voluntary subsidization could be a useful mechanism, where the broadband
still maintains a physically neutral network, while service differentiation is
maintained at a higher economic layer. Product differentiation is not uncommon
in other areas, e.g., first class airfare, student tickets, and etc. Our
on-going work [6] analyzes the subsidization competition among the content
providers when policy allows. Our preliminary results show that it could serve
a (physically) neutral and viable mechanism to attract investment incentives
for access ISPs. The danger is that new entrants in the edge provider market
might find it difficult to compete with established players with deeper
pockets.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">21.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para
40: The European ISPs have to unbundle their local loops, which creates
competition among the providers. Thus, even when there is no policy to prohibit
ISPs to block traffic, competition will drive them not to hurt user welfare, if
such a blocking does so. This is supported by our public option work [1] and
the developments in the UK market [7]. On the other hand, the lack of
competition in the US (because of no local-loop unbundling) makes policy more
important. The problem is not about whether there are cases of blocking, but
whether competition is enough for users to choose providers in case one of them
is not satisfactory to the users.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">22.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para
41: Again, the problem is related to the lack of competition. When AT&T is
the only provider and it lacks capacity, there might be a good reason (network
management, or economic) for it to restrict applications. It is difficult to
judge whether such a restriction is good or bad. Without such a restriction,
capacity-heavy application might affect the usage of other apps when the
wireless capacity is limited. When competition is there, users will have
choices to switch to another provider which might allow such apps, but probably
with poorer quality. In a monopoly case, regulation needs to be carefully
designed. If we want to guarantee a minimum service quality for apps, this
threshold might depend on the capacity of the provider and other factors.
Instead, we propose to limit the quality difference [5] among services rather
than putting an absolute guaranteed minimum service quality for the free
services.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">23.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para
42: we do not agree that “the threat of …. Did not depend on … market power”.
In fact, that is the root cause. We have shown the impact of market power in
our work [1, 3].<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">24.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para
43: we do not agree that “the commission need not engage in a market power
analysis to justify its rules …”.<span style="mso-spacerun: yes;"> </span>“the
ability to block edge providers depended on “end users not being fully
responsive to the imposition of such restrictions””, we comment that
transparency is needed to make users to be aware of such restrictions. The
no-responsiveness could be because the restriction does not affect users’
welfare/utility much and switching to another provider is not necessary. The
real threat is that it matters to the users, but due to the lack of market
competition and user choices, the users have to stay with the current provider.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">25.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
44: “broadband providers have incentives and the economic ability to limit
Internet openness …” we comment that broadband providers do not have a direct
incentive to limit openness, unless it is going to increase their
profit/revenue. In a competitive market, they won’t have such incentives if
users are sensitive to service quality. Also, broadband providers do not have
strong incentives to limit the openness on the end-user side, but have more
incentive to differentiate at the peering side with the edge providers, who
might have much larger profit margins compared to the access providers. The
changing market place, including the change in content, e.g. the rise of
Netflix and CDNs, has affected the broadband peering strategies greatly.
Technology-wise, it is not difficult to physically separate different capacities
along different routing paths. With the emerging Software Defined Networking (SDN)
approaches, ISPs might have stronger abilities to manage their network
resources to differentiate their peering relationships with other ISPs/CPs. Our
work [1] showed that by differentiate capacity at the CP-side, the ISP has
incentives to do so to increase revenue under a monopoly setting. In
particular, the ISP has incentives to make free/open class “damaged good” to
force CPs to use premium class and pay, which might hurt social welfare. We
showed that competition, e.g., the introduction of Public option, could solve
the problem. Justification for charging edge providers could be the basic
economic efficiency: capacity is limited, by differentiate high- and low-value
traffic, the ISP could provide higher welfare to users and CPs.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">26.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
45: Transit ISPs use parameters in BGP routing protocol to choose different
routing paths. Access ISPs use public/private peering settings to control
traffic. On the user-side, mobile providers use data cap to control and price
traffic data. AT&T’s sponsored data plan introduces voluntary subsidization
from CPs to end-users for their over-cap traffic. As for the Comcast case, its
private peering practice might be used to exploit their monopoly market power
in the access market. Level3 data hints at that.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">27.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
46: we do not have data to understand the switching costs. However, we support
that transparency to be given to users, so that they understand more of their
choices/options and improve the market competition.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">28.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
47: “We seek comment on the state of competition in broadband Internet access
service, and its effect on providers’ incentives to limit openness.” We guess
many people will comment on the monopolistic status of Comcast. Our public
option work [1] showed that the monopoly has incentives to differentiate
service at the CP-side and show its incentives to make public peering “damaged
good”, which is claimed by Level3. Thus, we advocate a public option approach
to introduce competition, under which no regulation is needed to monopoly.
Otherwise, suitable regulation might be needed to regulate the monopoly. DSL’s
openness practice could incentivize high-speed broadband to differentiate CPs
less aggressive, although another high-speed provider’s competition will help
more.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">29.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
48: needs more market data for market competition analysis.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">30.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
49: as our study [1] showed, market competition is a key to whether regulation
is needed. We suggest that regulation should be closely coupled with market
power analysis.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">31.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
50: “there are other economic theories …” 1) Shapley value for inter-ISP
settlement [3, 4], and 2) public option [1] for access ISP service
differentiation and competition, and 3) subsidization competition [6] among the
content providers. Also, on the value of the network, the exact form of n^2 or
n log n doesn’t matter - the important theoretical distinction is whether the
value of the network is a convex function of n or not - and both n^2 and
n log n satisfy convexity.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">32.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
51: technically, access ISP might be able to block end-user traffic; however,
flow-based control is expensive and difficult for ISPs, and has little economic
incentives to do so. ISPs largely use public/private peering nowadays to
differentiate classes of traffic via inter-AS routing. The routing is currently
based on BGP protocol, a rough tool to do fine grain flow-level control. Thus,
we suggest the rules extend its scope to include inter-AS settlement and
peering relationships.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">33.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
52: yes, ISPs could use traffic management tools. However, most practices are
simply macro-level private/peering relationships, where physical links are
separated/isolated and no active differentiations are needed for the traffic
flows.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">34.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
59: “.. the rules were not intended “to affect existing arrangements for
network interconnection, including existing paid peering arrangements””, we
comment that the inter-connect agreement between the broadband and other
transit ISPs should be included within the scope, otherwise, less could be done
to limit the service differentiation done on the CP-side.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">35.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
60: in general, a “specialized service” could be a prioritized service which
brings extra revenue for the ISP and its users. However, as we showed in our
work [1], ISPs might have incentives to make ordinary service “damaged good” so
as to force users to use specialized service and pay them. Thus, we might need
to impose conditions under which specialized service could be provided.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">36.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
62: this is reasonable and useful to distinguish mobile and fixed service
providers, and based on their characteristics, e.g., cost structure and
capacity, to impose different regulations, e.g., different minimum requirement
for service and thresholds for service quality differences.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">37.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
63-88: we general support the transparency in any aspect, as long as it’s
practically feasible. The main reason is that transparency could improve market
completion, under which strict regulations might not even be needed.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">38.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
89-109: blocking is definitely undesirable for users and CPs. However, the D.C.
Circuit vacated the rule mainly because of the legal “common carriage”
requirement, which the ISPs might not be responsible for. We do not oppose the
proposed rule, however, it still does not have a legal justification and “a
clarification … does not preclude …” makes the rule almost non-enforcing. To
impose a minimum level of service is encouraging; however, the legal
justification is still missing and what this level to set needs further
thoughts.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">39.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
90: “relationship between no-blocking and commercially reasonable rules”.
No-blocking is a special case of the later. <o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">40.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
101: “we seek comment on how minimum level of access should be defined”. This
is important. Different type of ISPs should have different thresholds. Different
type of CPs might have different minimum requirements, although from a
regulatory and practical perspective, it is difficult to set different
requirements for each different CP. Furthermore, under sufficient competition,
we think even the minimum requirements are not needed. <o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">41.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
104: we agree that the requirement should also be evolving based on the
changing characteristics of content and user expectations.<o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">42.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
110-141: “Codifying an enforceable rule that is not common carriage per se”. <span style="mso-spacerun: yes;"> </span>D.C. rejected prior rule based on “(it) so
limited broadband providers’ control over edge providers’ transmissions that
[it] constituted common carriage per se.” from this perspective, I feel that a
requirement of minimum service requirement is still like imposing a “common
carriage” requirement on the ISPs, which does not have a legal justification
for the FCC (although it might have its economic justifications). Therefore, in
our work [5], we suggest a milder/weak, but enforceable rule, which restrict
the difference in service qualities of the different services provided by the
ISPs. If the ISP is really capacity constrained, then its premium service
quality cannot be high, unless its ordinary service has to be maintained at
certain level. The advantage is that it does not impose a fixed/hard
requirement on the ISPs, and the rule is flexible for different types of
providers, e.g., mobile provider might maintain lower QoS for ordinary (e.g.,
best-effort) and it premium service (e.g., 1Mbps), while high-speed broadband
can provide very high-quality premium service (e.g., 10Mbps), when its ordinary
service can guarantee already quite good service (e.g. 2Mbps). By imposing such
a relative regulation, it does not impose any hard requirement for ISPs to
fulfill. In comparison, minimum requirement rule read more like “you have to
become a common carriage first, so as to be qualified to differentiate
services”. Both have similarities and differences in terms of justifications
and practicality. <o:p></o:p></div>
<div class="MsoListParagraphCxSpMiddle" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">43.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
142-160: these are legal authority and considerations, not our expertise. We
suppose the rule needs to avoid imposing “common carriage” kind of regulations
to ISPs.<o:p></o:p></div>
<div class="MsoListParagraphCxSpLast" style="mso-list: l0 level1 lfo1; text-indent: -18.0pt;">
<!--[if !supportLists]--><span style="mso-bidi-font-family: Calibri; mso-bidi-theme-font: minor-latin; mso-fareast-font-family: Calibri; mso-fareast-theme-font: minor-latin;"><span style="mso-list: Ignore;">44.<span style="font: 7.0pt "Times New Roman";"> </span></span></span><!--[endif]-->Para.
161-176: this is dispute resolution. We comment that our proposed Shapley value
mechanism can be used for inter-AS dispute resolution. However, it is more on
the peering side, not on the end-user side.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
References<o:p></o:p></div>
<div class="MsoNormal">
[1] Richard T. B. Ma and Vishal Misra. <b style="mso-bidi-font-weight: normal;">The Public Option: A Non-Regulatory Alternative to Network Neutrality.</b>
<i style="mso-bidi-font-style: normal;">IEEE/ACM Transactions on Networking,
Volume 21, Issue 6, pp. 1866 - 1879, December, 2013.<o:p></o:p></i></div>
<div class="MsoNormal">
[2] Jingjing Wang, Richard T. B. Ma and Dah Ming Chiu.<span style="mso-spacerun: yes;"> </span><b>Paid Prioritization and Its Impact on Net
Neutrality. </b><i style="mso-bidi-font-style: normal;"><span style="mso-bidi-font-weight: bold;">Proceedings of</span></i><b> </b><em><span style="font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">IFIP
Networking Conference 2014.<o:p></o:p></span></em></div>
<div class="MsoNormal">
[3] Richard T. B. Ma, Dah Ming Chiu, John C. S. Lui, Vishal
Misra and Dan Rubenstein. <b style="mso-bidi-font-weight: normal;">On Cooperative
Settlement Between Content, Transit and Eyeball Internet Service Providers.</b>
<i style="mso-bidi-font-style: normal;">IEEE/ACM Transactions on Networking,
Volume 19, Issue 3, pp. 802 - 815, June, 2011.<o:p></o:p></i></div>
<div class="MsoNormal">
[4] Richard T. B. Ma, Dah Ming Chiu, John C. S. Lui, Vishal
Misra and Dan Rubenstein. <b style="mso-bidi-font-weight: normal;">Internet
Economics: The Use of Shapley Value for ISP Settlement.</b><i style="mso-bidi-font-style: normal;"> IEEE/ACM Transactions on Networking, Volume 18, Issue 3, pp. 775 -
787, June, 2010.<o:p></o:p></i></div>
<div class="MsoNormal">
[5] Jing Tang and Richard T. B. Ma. <b style="mso-bidi-font-weight: normal;">Regulating Monopolistic ISPs without Neutrality.</b> <i style="mso-bidi-font-style: normal;"><span style="mso-bidi-font-weight: bold;">Proceedings
of</span></i><b> </b><em><span style="font-family: Calibri; mso-ascii-theme-font: minor-latin; mso-hansi-theme-font: minor-latin;">IEEE International Conference on
Networking Protocols (ICNP) 2014.<o:p></o:p></span></em></div>
<div class="MsoNormal">
[6] Richard T. B. Ma. <b style="mso-bidi-font-weight: normal;">Subsidization
Competition: Vitalizing the Neutral Internet</b>. <i style="mso-bidi-font-style: normal;">Working paper. </i><a href="http://arxiv.org/pdf/1406.2516v1.pdf"><i style="mso-bidi-font-style: normal;">http://arxiv.org/pdf/1406.2516v1.pdf</i></a><i style="mso-bidi-font-style: normal;">. <o:p></o:p></i></div>
<div class="MsoNormal">
<span style="mso-bidi-font-style: italic;">[7]<i> </i>Thomas
Newton.<i> </i><b style="mso-bidi-font-weight: normal;">ISP Traffic Management:
BT vs Virgin vs Sky vs TalkTalk vs EE. </b></span><span style="mso-field-code: " HYPERLINK \0022http\:\/\/recombu\.com\/digital\/news\/isp-traffic-management-bt-sky-virgin-media-ee-talktalk_M11045\.html\0022 \\t \0022_blank\0022 ";"><span class="MsoHyperlink"><i style="mso-bidi-font-style: normal;">http://recombu.com/digital/news/isp-traffic-management-bt-sky-virgin-media-ee-talktalk_M11045.html</i></span></span><i style="mso-bidi-font-style: normal;"><o:p></o:p></i></div>
<!--[if gte mso 9]><xml>
<o:DocumentProperties>
<o:Revision>0</o:Revision>
<o:TotalTime>0</o:TotalTime>
<o:Pages>1</o:Pages>
<o:Words>3570</o:Words>
<o:Characters>20351</o:Characters>
<o:Company>Columbia University</o:Company>
<o:Lines>169</o:Lines>
<o:Paragraphs>47</o:Paragraphs>
<o:CharactersWithSpaces>23874</o:CharactersWithSpaces>
<o:Version>14.0</o:Version>
</o:DocumentProperties>
<o:OfficeDocumentSettings>
<o:AllowPNG/>
</o:OfficeDocumentSettings>
</xml><![endif]-->
<!--[if gte mso 9]><xml>
<w:WordDocument>
<w:View>Normal</w:View>
<w:Zoom>0</w:Zoom>
<w:TrackMoves/>
<w:TrackFormatting/>
<w:PunctuationKerning/>
<w:ValidateAgainstSchemas/>
<w:SaveIfXMLInvalid>false</w:SaveIfXMLInvalid>
<w:IgnoreMixedContent>false</w:IgnoreMixedContent>
<w:AlwaysShowPlaceholderText>false</w:AlwaysShowPlaceholderText>
<w:DoNotPromoteQF/>
<w:LidThemeOther>EN-US</w:LidThemeOther>
<w:LidThemeAsian>ZH-CN</w:LidThemeAsian>
<w:LidThemeComplexScript>X-NONE</w:LidThemeComplexScript>
<w:Compatibility>
<w:BreakWrappedTables/>
<w:SnapToGridInCell/>
<w:WrapTextWithPunct/>
<w:UseAsianBreakRules/>
<w:DontGrowAutofit/>
<w:SplitPgBreakAndParaMark/>
<w:EnableOpenTypeKerning/>
<w:DontFlipMirrorIndents/>
<w:OverrideTableStyleHps/>
<w:UseFELayout/>
</w:Compatibility>
<m:mathPr>
<m:mathFont m:val="Cambria Math"/>
<m:brkBin m:val="before"/>
<m:brkBinSub m:val="--"/>
<m:smallFrac m:val="off"/>
<m:dispDef/>
<m:lMargin m:val="0"/>
<m:rMargin m:val="0"/>
<m:defJc m:val="centerGroup"/>
<m:wrapIndent m:val="1440"/>
<m:intLim m:val="subSup"/>
<m:naryLim m:val="undOvr"/>
</m:mathPr></w:WordDocument>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:LatentStyles DefLockedState="false" DefUnhideWhenUsed="true"
DefSemiHidden="true" DefQFormat="false" DefPriority="99"
LatentStyleCount="276">
<w:LsdException Locked="false" Priority="0" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Normal"/>
<w:LsdException Locked="false" Priority="9" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="heading 1"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 2"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 3"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 4"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 5"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 6"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 7"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 8"/>
<w:LsdException Locked="false" Priority="9" QFormat="true" Name="heading 9"/>
<w:LsdException Locked="false" Priority="39" Name="toc 1"/>
<w:LsdException Locked="false" Priority="39" Name="toc 2"/>
<w:LsdException Locked="false" Priority="39" Name="toc 3"/>
<w:LsdException Locked="false" Priority="39" Name="toc 4"/>
<w:LsdException Locked="false" Priority="39" Name="toc 5"/>
<w:LsdException Locked="false" Priority="39" Name="toc 6"/>
<w:LsdException Locked="false" Priority="39" Name="toc 7"/>
<w:LsdException Locked="false" Priority="39" Name="toc 8"/>
<w:LsdException Locked="false" Priority="39" Name="toc 9"/>
<w:LsdException Locked="false" Priority="35" QFormat="true" Name="caption"/>
<w:LsdException Locked="false" Priority="10" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Title"/>
<w:LsdException Locked="false" Priority="1" Name="Default Paragraph Font"/>
<w:LsdException Locked="false" Priority="11" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtitle"/>
<w:LsdException Locked="false" Priority="22" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Strong"/>
<w:LsdException Locked="false" Priority="20" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Emphasis"/>
<w:LsdException Locked="false" Priority="59" SemiHidden="false"
UnhideWhenUsed="false" Name="Table Grid"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Placeholder Text"/>
<w:LsdException Locked="false" Priority="1" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="No Spacing"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 1"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 1"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 1"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 1"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 1"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 1"/>
<w:LsdException Locked="false" UnhideWhenUsed="false" Name="Revision"/>
<w:LsdException Locked="false" Priority="34" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="List Paragraph"/>
<w:LsdException Locked="false" Priority="29" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Quote"/>
<w:LsdException Locked="false" Priority="30" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Quote"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 1"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 1"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 1"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 1"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 1"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 1"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 1"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 1"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 2"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 2"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 2"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 2"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 2"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 2"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 2"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 2"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 2"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 2"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 2"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 2"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 2"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 2"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 3"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 3"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 3"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 3"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 3"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 3"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 3"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 3"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 3"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 3"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 3"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 3"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 3"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 3"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 4"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 4"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 4"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 4"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 4"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 4"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 4"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 4"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 4"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 4"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 4"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 4"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 4"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 4"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 5"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 5"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 5"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 5"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 5"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 5"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 5"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 5"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 5"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 5"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 5"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 5"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 5"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 5"/>
<w:LsdException Locked="false" Priority="60" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Shading Accent 6"/>
<w:LsdException Locked="false" Priority="61" SemiHidden="false"
UnhideWhenUsed="false" Name="Light List Accent 6"/>
<w:LsdException Locked="false" Priority="62" SemiHidden="false"
UnhideWhenUsed="false" Name="Light Grid Accent 6"/>
<w:LsdException Locked="false" Priority="63" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 1 Accent 6"/>
<w:LsdException Locked="false" Priority="64" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Shading 2 Accent 6"/>
<w:LsdException Locked="false" Priority="65" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 1 Accent 6"/>
<w:LsdException Locked="false" Priority="66" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium List 2 Accent 6"/>
<w:LsdException Locked="false" Priority="67" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 1 Accent 6"/>
<w:LsdException Locked="false" Priority="68" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 2 Accent 6"/>
<w:LsdException Locked="false" Priority="69" SemiHidden="false"
UnhideWhenUsed="false" Name="Medium Grid 3 Accent 6"/>
<w:LsdException Locked="false" Priority="70" SemiHidden="false"
UnhideWhenUsed="false" Name="Dark List Accent 6"/>
<w:LsdException Locked="false" Priority="71" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Shading Accent 6"/>
<w:LsdException Locked="false" Priority="72" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful List Accent 6"/>
<w:LsdException Locked="false" Priority="73" SemiHidden="false"
UnhideWhenUsed="false" Name="Colorful Grid Accent 6"/>
<w:LsdException Locked="false" Priority="19" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Emphasis"/>
<w:LsdException Locked="false" Priority="21" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Emphasis"/>
<w:LsdException Locked="false" Priority="31" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Subtle Reference"/>
<w:LsdException Locked="false" Priority="32" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Intense Reference"/>
<w:LsdException Locked="false" Priority="33" SemiHidden="false"
UnhideWhenUsed="false" QFormat="true" Name="Book Title"/>
<w:LsdException Locked="false" Priority="37" Name="Bibliography"/>
<w:LsdException Locked="false" Priority="39" QFormat="true" Name="TOC Heading"/>
</w:LatentStyles>
</xml><![endif]-->
<!--[if gte mso 10]>
<style>
/* Style Definitions */
table.MsoNormalTable
{mso-style-name:"Table Normal";
mso-tstyle-rowband-size:0;
mso-tstyle-colband-size:0;
mso-style-noshow:yes;
mso-style-priority:99;
mso-style-parent:"";
mso-padding-alt:0cm 5.4pt 0cm 5.4pt;
mso-para-margin-top:0cm;
mso-para-margin-right:0cm;
mso-para-margin-bottom:10.0pt;
mso-para-margin-left:0cm;
line-height:115%;
mso-pagination:widow-orphan;
font-size:11.0pt;
font-family:Calibri;
mso-ascii-font-family:Calibri;
mso-ascii-theme-font:minor-latin;
mso-hansi-font-family:Calibri;
mso-hansi-theme-font:minor-latin;
mso-fareast-language:ZH-CN;}
</style>
<![endif]-->
<!--StartFragment-->
<!--EndFragment--><br />
<div class="MsoNormal">
<br /></div>
</div>
Vishal Misrahttp://www.blogger.com/profile/04748385949397825305noreply@blogger.com2tag:blogger.com,1999:blog-4160882474275537188.post-13203860780542971382014-06-11T11:15:00.000-07:002014-06-12T09:11:22.378-07:00How much competition is enough?<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Arial, Helvetica, sans-serif;">The topic of competition is a very interesting one and with layers and layers of complexity. While I firmly believe that the path to an Open Internet goes through building competition at the broadband level, and <i>not</i> through complex network neutrality regulations, the issue is not simple.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">On the one hand, we have the example of UK, where OfCom has kept a pretty much<a href="http://stakeholders.ofcom.org.uk/consultations/net-neutrality/statement/" target="_blank"> hands off approach to Network Neutrality</a>, the result has been <a href="http://recombu.com/digital/news/isp-traffic-management-bt-sky-virgin-media-ee-talktalk_M11045.html" target="_blank">a very open Internet</a> that is market and competition driven. Almost every consumer in the UK has access to 4 ISPs that are similar in terms of capabilities and the Internet has remained "neutral". On the other hand we have the example in the US, where although significant portions of the nation have at least two comparable broadband providers (say, metropolitan DC and New Jersey where Comcast and Verizon FiOS are widely available) the "openness of the Internet" has been a problem. Specifically there have been <a href="http://online.wsj.com/news/articles/SB10001424052702304899704579391223249896550" target="_blank">peering disputes between Netflix and broadband providers</a>, or the CDNs employed by Netflix and broadband providers. Netflix and Level3 have been <a href="http://blog.level3.com/global-connectivity/chicken-game-played-child-isps-internet/" target="_blank">complaining loudly</a> that this is coercion from the ISP and quality is being deliberately deteriorated to extract a toll from the content providers. So the question is - if 4 competitors provide Internet openness in the UK, why do 2 providers fail to do so in the US? What gives the broadband providers in the US the confidence to flex their muscles, whereas similar profit oriented entities in the UK don't?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">The answer to that involves a lot of factors. First off is the question of long term contracts - in the UK they are prohibited as far as I know, whereas in the US they are commonplace. This makes "competition" less than perfect. Secondly, since the broadband providers in the US are also invariably cable providers, there are vertically integrated monopolies in the US that are absent in the UK. There are many factors that make an ISP sticky for a consumer. However, the point that I wanted to bring across in this post is what happens when there is "true" competition - is there a <i>magic number</i> of competitors (4 vs 2) that ensures openness and "good behavior" by ISPs? Isn't 2 providers competing the same as 4 providers competing?</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">The answer to that is no. Competition is <i>monotonically</i> better for consumers. 3 providers competing is better than 2, and 4 is better than 3. And this is with the assumption that there is no collusion etc. happening with a smaller number of competitors, i.e. the best case scenario for competition. In a prior work, we applied cooperative game theory techniques to analyze the Internet ecosystem:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /><span style="background-color: white; line-height: 18px;">Richard T.B. Ma, Dahming Chiu, John C.S. Lui, Vishal Misra and Dan Rubenstein</span><span style="background-color: white; color: #666666; font-size: 12px; line-height: 18px;">,</span> <a href="http://dna-pubs.cs.columbia.edu/citation/paperfile/172/CoNEXT08.pdf" target="_blank">On Cooperative Settlement Between Content, Transit and Eyeball Internet Service Providers, Proceedings of 2008 ACM Conference on Emerging network experiment and technology (CoNEXT 2008), Madrid, Spain, December, 2008</a></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />The associated talk we gave is <a href="http://dna-pubs.cs.columbia.edu/citation/presentationfile/172/CoNext08.ppt" target="_blank">here</a>, but let me try to explain some numbers around competition that can explain the behavior we have observed.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span><span style="font-family: Arial, Helvetica, sans-serif;">An important concept of cooperative game theory is the Shapley value, which is the share that an individual gets of the value generated by the coalition it is part of. Shapley value incorporates various factors like the value an individual brings to the coalition, the value of the coalition without that individual etc. and the end formula gives guidance on what a rational individual would do to maximize it's share under all scenarios. </span><span style="font-family: Arial, Helvetica, sans-serif;">Our work has the following formula of the “fair share” (the Shapley value) of the class of providers in the economic ecosystem of the Internet:</span><br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Let’s say there are m content providers and n ISPs. Then the share of the value generated (V) for the content providers is n/(m*(n+m)) and that of the ISPs is m/(n*(n+m)).</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">Let’s say Netflix is the sole content provider (m=1) at one end and there are 2 ISPs competing for customers (n = 2). Then Netflix’s share is 2/3 and each ISP is 1/6. So for every say 12 dollars a customer generates for Netflix every month, the ISPs have reason to believe that they deserve 2 dollars of it, because Netflix's business wouldn't exist without the ISPs (I am deliberately leaving out arguments of broadband being a utility that every consumer has the right to etc.).</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">If now the number of ISPs competing becomes 3, the Netflix share becomes 3/4 and the ISP share is 1/12 each. If you move to 4 ISPs, then Netflix share becomes 4/5, and each ISP is 1/20. Now for the 12 dollars that Netflix generates, the ISPs believe they deserve 60 cents from it.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;">So moving from 2 to 4 the “rightful share” of the value generated reduces to 1/20th from 1/6th - it is plausible that it is not worth it at that point to play hardball and extract that revenue from Netflix and instead the ISPs are more interested in winning and keeping customers. Maybe at 1/6th (2 ISPs competing) it is worth it to lose a few customers if you end up extracting more from the content provider but increasing the level of competition reduces the utility of that tactic. At some point the expected payoff (of toll) from the content provider falls below the expected loss (of customer revenue) to competing ISPs and at that point "competition is enough".</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">So more competition is better for consumers, and the cooperative game theory analysis provides some numbers to reason about <i>how</i> much better.</span></div>
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<div>
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span></div>
</div>
Vishal Misrahttp://www.blogger.com/profile/04748385949397825305noreply@blogger.com5tag:blogger.com,1999:blog-4160882474275537188.post-62017795115046612252014-04-28T07:56:00.002-07:002014-04-28T11:05:49.377-07:00How should consumers benefit if Netflix adopts P2P technology?<div dir="ltr" style="text-align: left;" trbidi="on">
Over the weekend there was considerable chatter on the Internet about Netflix and Peer-to-Peer (P2P) architectures. According to news reports like <a href="http://arstechnica.com/information-technology/2014/04/netflix-researching-large-scale-peer-to-peer-technology-for-streaming/" target="_blank">this</a>, Netflix is exploring P2P technology as a delivery mechanism, to defray cost of delivery to consumers. This becomes even relevant with the new net neutrality proposals that will allow "<a href="http://money.cnn.com/2014/04/23/technology/open-internet/" target="_blank">fast lanes</a>" to exist on the Internet, paving the way for ISPs to charge more from customers like Netflix which will then ultimately get passed on to Netflix's consumers.<br />
<br />
However, if Netflix does intend to use P2P to work around this incoming increase in costs, why should consumers help out Netflix? We tackled this precise issue a few years ago and provided some answers. We looked at scenarios of a coalition that has one large "atomic" player (e.g. Netflix) and infinitesimally small peers (relative to the atomic player). The idea is applicable for many different scenarios, for example <a href="http://en.wikipedia.org/wiki/Femtocell" target="_blank">Femtocells</a>, the <a href="http://en.wikipedia.org/wiki/FON" target="_blank">FON</a> network, P2P peer-assisted streaming the kind Netflix is exploring etc. We asked the question: how should the coalition profits (or cost savings) be fairly distributed amongst its constituents? We applied coalition game theory and developed new results for fluid Shapley values and the following paper was published:<br />
<br />
<span style="background-color: white; color: #666666; font-family: verdana, arial, helvetica, sans-serif; font-size: 12px; line-height: 18px;">Vishal Misra, Stratis Ioannidis, Augustin Chaintreau and Laurent Massoulie, </span><b style="background-color: white; color: #666666; font-family: verdana, arial, helvetica, sans-serif; font-size: 12px; line-height: 18px;"><a href="http://dna-pubs.cs.columbia.edu/citation/paperfile/185/incentive.pdf" target="_blank">Incentivizing Peer-Assisted Services: A Fluid Shapley Value Approach</a></b><span style="background-color: white; color: #666666; font-family: verdana, arial, helvetica, sans-serif; font-size: 12px; line-height: 18px;">, </span><i style="background-color: white; color: #666666; font-family: verdana, arial, helvetica, sans-serif; font-size: 12px; line-height: 18px;">Proceedings of ACM/Sigmetrics</i><span style="background-color: white; color: #666666; font-family: verdana, arial, helvetica, sans-serif; font-size: 12px; line-height: 18px;">, New York, June, 2010. </span><br />
<br />
The presentation associated with the paper can be obtained <a href="http://dna-pubs.cs.columbia.edu/citation/presentationfile/185/FluidShapley.pptx" target="_blank">here</a>.<br />
<br />
The paper has all the details but the result applicable to the scenario of peer-assisted streaming would be that consumers should get back roughly <i>half</i> of the dollars they save Netflix in terms of bandwidth costs. Say a customer of Netflix uploads 10GB a month on behalf of Netflix to other customers. If Netflix had instead paid a dollar to it's CDN for this 10GB (or the ISP for the "fast lane access"), then assuming a linear cost model this customer should be getting 50 cents back from Netflix for the help that it provides. The paper has complete analysis that includes general (non-linear) cost models if it interests you but the message is that if providers switch to peer-assistance, then the peers need to be compensated appropriately. Potentially this is a win-win solution for all (perhaps not for the "fast-lane charging" ISPs!).</div>
Vishal Misrahttp://www.blogger.com/profile/04748385949397825305noreply@blogger.com0tag:blogger.com,1999:blog-4160882474275537188.post-50990599876047839082014-02-24T09:48:00.000-08:002014-02-25T07:02:31.820-08:00Why is Netflix paying Comcast?<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Our group has been researching and thinking about Internet Economics for quite sometime now. <a href="http://online.wsj.com/news/articles/SB10001424052702304834704579401071892041790" target="_blank">Events</a> are backing up our predictions, and they are not good for consumers.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Over 5 years ago, </span><span style="font-family: Arial, Helvetica, sans-serif;">we published the following work, </span><span style="font-family: Arial, Helvetica, sans-serif;">driven by at that time a PhD student of ours, </span><a href="https://www.comp.nus.edu.sg/~tbma/" style="font-family: Arial, Helvetica, sans-serif;" target="_blank">Richard Ma</a><span style="font-family: Arial, Helvetica, sans-serif;">: </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="background-color: white; font-family: Arial, Helvetica, sans-serif; line-height: 18px;">Richard T.B. Ma, Dahming Chiu, John C.S. Lui, Vishal Misra and Dan Rubenstein</span><span style="background-color: white; color: #666666; font-family: Arial, Helvetica, sans-serif; font-size: 12px; line-height: 18px;">,</span><span style="font-family: Arial, Helvetica, sans-serif;"> </span><a href="http://dna-pubs.cs.columbia.edu/citation/paperfile/172/CoNEXT08.pdf" style="font-family: Arial, Helvetica, sans-serif;" target="_blank">On Cooperative Settlement Between Content, Transit and Eyeball Internet Service Providers, Proceedings of 2008 ACM Conference on Emerging network experiment and technology (CoNEXT 2008), Madrid, Spain, December, 2008</a><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">I am happy to answer any questions regarding the math/analysis in the paper, and the slides associated with the talk we gave are </span><a href="http://dna-pubs.cs.columbia.edu/citation/presentationfile/172/CoNext08.ppt" style="font-family: Arial, Helvetica, sans-serif;" target="_blank">here</a><span style="font-family: Arial, Helvetica, sans-serif;">, but in the next few lines I will try and explain the </span><i style="font-family: Arial, Helvetica, sans-serif;">implications</i><span style="font-family: Arial, Helvetica, sans-serif;"> of the analysis.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">We looked at the economic ecosystem of the Internet, using Shapley values, with cooperative game theory as the guiding principle. Our analysis revealed something interesting: it showed that the early Internet settlement model was stable and made sense. The Internet was composed of light email and web traffic between essentially symmetrical end points like educational and research institutions. ISPs were two kinds - eyeball and transit (or Tier-1). Transit ISPs provided global connectivity, and t</span><span style="font-family: Arial, Helvetica, sans-serif;">he eyeball ISPs bought bandwidth from transit ISPs. Consumers bought connectivity from the eyeball ISPs and interacted with each other, and buying bandwidth was natural. By doing bilateral settlements between the parties directly interacting, a stable solution can be arrived at. The solution lies in what is called in cooperative game theory as the </span><i style="font-family: Arial, Helvetica, sans-serif;"><a href="http://en.wikipedia.org/wiki/Core_(game_theory)" target="_blank">core</a></i><span style="font-family: Arial, Helvetica, sans-serif;">.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Things started to change when the Internet e-Commerce system started developing. Well known content providers started to emerge. The ISPs that sold bandwidth to content providers were classified as content ISPs, and CDNs can be thought of as another form of </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">content ISPs. Now the Internet was generating two revenue streams - one that was provided by selling connectivity/bandwidth to the consumers and content providers, and the second was the revenue generated by the e-Commerce activities of content providers like Google, Amazon, eBay, Netflix. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Our analysis showed that if the two revenue streams differed significantly in magnitude, then the current settlement model of the Internet was <i>unstable.</i> Specifically, no model that followed the simple customer-provider of bandwidth settlement model could lead to a stable solution and the solution was always outside the core. If the eCommerce revenue stream is much greater than the revenue stream from bandwidth connectivity, then the eyeball ISPs are in fact subsidizing the content providers. The eyeball ISPs should not be <i>buying </i>bandwidth from upstream providers, but should rather be selling access to the eyeballs that they are providing connectivity to. It predicted a reverse customer/provider relationship to emerge as well as <i>paid-peering</i> which is what the recent <a href="http://gigaom.com/2014/02/23/confirmed-comcast-and-netflix-have-signed-a-peering-agreement/" target="_blank">Netflix/Comcast arrangement</a> is being called. </span><br />
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: Arial, Helvetica, sans-serif; text-align: left;"><br /></span></div>
<div class="separator" style="clear: both; text-align: justify;">
<span style="font-family: Arial, Helvetica, sans-serif; text-align: left;">If the converse is true, and the eyeball ISPs are generating a lot more revenue than the e-Commerce revenue, then again a stable solution lies in the settlement model where revenue is flowing to the content side of the equation. The eyeball ISPs should be paying the content providers to generate content, so they can sell connectivity to the consumers that want content. </span></div>
<div style="text-align: left;">
<br /></div>
<span style="font-family: Arial, Helvetica, sans-serif;">There is another form of asymmetry that makes the issue of asymmetry of revenue almost moot. And that is the asymmetry of market power, and it follows directly from our analysis. If there is no competition on the content side but there is competition on the eyeball side, then the content provider has the leverage to extract better terms for the eyeball ISP. </span><span style="font-family: Arial, Helvetica, sans-serif;">If however, there is competition on the </span><i style="font-family: Arial, Helvetica, sans-serif;">content</i><span style="font-family: Arial, Helvetica, sans-serif;"> side but not on the eyeball that increases the leverage that the eyeball side has on the content side. Consumers who have no choice for broadband ISPs are in some ways trapped and then it is <i>rational</i> for content providers to pay the eyeball providers to keep earning that revenue (which can then be passed back to the consumer of course). Consumers won't switch off broadband because Netflix is slow - there is Amazon, iTunes, Hulu just on the video side of things, and of course there is countless other "competition" that the Internet provides in the form of content. If the eyeball ISP is a monopoly, it will <i>keep </i>its business, Netflix performing well or not.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">Our analysis predicted this, and since this is simply "paid peering", it is in no violation of network neutrality even if that concept existed and was backed by courts. This is the only way by which a solution ends up being in the core (i.e., it is stable).</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif;">The problem really is the monopoly that last mile providers have, and it cannot be fixed by regulation - only by competition. The possibility of competition in the broadband market in the US was further reduced by the announcement of the Time Warner "Cable" acquisition by Comcast "Cable". That was the subject of some follow on work that Richard and I did, and I have blogged about it <a href="http://peerunreviewed.blogspot.com/2014/02/the-public-option-municipal-broadband.html" target="_blank">here</a>. It is also the issue that <a href="http://scrawford.net/" target="_blank">Susan Crawford</a> has spoken about extensively, most notably in her book </span><span style="font-family: Arial, Helvetica, sans-serif;">Captive Audience: The Telecom Industry & Monopoly Power in the New Gilded Age.</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /></span>
<br />
<br /></div>
Vishal Misrahttp://www.blogger.com/profile/04748385949397825305noreply@blogger.com0tag:blogger.com,1999:blog-4160882474275537188.post-61579058615227217862014-02-16T12:31:00.000-08:002014-05-12T09:11:59.688-07:00The Public Option: Municipal Broadband Access <div dir="ltr" style="text-align: left;" trbidi="on">
The recent announcement of the <a href="http://money.cnn.com/2014/02/13/technology/comcast-time-warner-cable-deal/" target="_blank">Comcast acquisition of Time Warner Cable</a> has caused quite a furore. Consumer advocates are worried that it will cause problems, higher prices, bad performance etc. The issue is it will lead to the creation of a giant monopoly. Comcast argues that it is not a monopoly and as it is Time Warner and Comcast do not compete in any market so it should make no difference anyway. There are two issues here: (a) first is the question why have Time Warner and Comcast, two giant corporations <i>not</i> competed anywhere yet? and, (b) the second and more important issue is not that of competition for cable service, which ostensibly Dish Networks also provides, but for Broadband access, where 19 of the top 20 metropolitan areas in the US will have only <i>one</i> choice for wired Broadband.<br />
<br />
The merger/acquisition is not between two <i>Cable</i> companies, but that of wired <i>Broadband ISPs</i> that are virtual monopolies.<br />
<br />
Our work, <a href="http://dna-pubs.cs.columbia.edu/citation/paperfile/212/ToN_PublicOption.pdf" style="font-family: verdana, arial, helvetica, sans-serif; font-size: 12px; line-height: 18px;" target="_blank">Richard T. B. Ma and Vishal Misra, <b>The Public Option: a Non-regulatory Alternative to Network Neutrality</b>, <i>IEEE/ACM Transactions on Networking</i>, 2013</a>, proposed a non-regulatory option to Network Neutrality that works for consumers. A presentation based on the work is available <a href="http://dna-pubs.cs.columbia.edu/citation/presentationfile/200/PublicOption.pdf" target="_blank">here</a>. We developed a realistic equilibrium model for congestion on the Internet, taking into account content bandwidth demand, user patience levels and the impact of protocols like TCP.<br />
<br />
Our analysis showed that if the Internet provider is a monopoly, it can hurt consumers in many ways. If Network Neutrality is not in place, ISPs can create scenarios that hurt both consumers as well as content providers, with the remedy being paying more revenue to the ISP. This revenue can be collected directly from the broadband subscribers, or by the content providers in the style of <a href="http://www.tomsguide.com/us/sponsored-data-faq,news-18200.html" target="_blank">AT&T sponsored data</a>. In either case, the extra costs will be passed on to the consumer.<br />
<br />
The alternative is <i>true</i> competition with multiple providers, where consumers have options that they can <i>instantly</i> switch to (and for our arguments, a month qualifies as instantly). In fact, our work shows that if true competition exists with multiple providers, network neutrality does <i>not </i>benefit consumers and there is a case to be made for non-neutral ISP services, one size does not fit all. But only if there is true competition. A way to get more competition at the eyeball level for consumers is via something like <a href="http://en.wikipedia.org/wiki/Local-loop_unbundling" target="_blank">local loop unbundling</a> (LLU). LLU operators are prevalent in Europe where the broadband market does have healthy competition and lower rates than the US.<br />
<br />
The other option is what we call The Public Option. The Public Option is an ISP that <i>voluntarily</i> implements Network Neutrality. Our work shows that the presence of the Public Option is good for social welfare - ISPs can implement non network-neutral policies if they desire, but the consumers and content providers do not lose out.<br />
<br />
One of the possibilities that we laid out in our paper was that The Public Option be implemented via municipal broadband.<br />
<br />
Our proposal is almost exactly what <a href="http://scrawford.net/" target="_blank">Susan Crawford</a> has been advocating, like the <a href="http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2366044" target="_blank">municipal broadband</a> effort in Massachusetts. As she spoke about in a recent talk, the concept has shown to be viable economically in Stockholm, via the <a href="http://www.stokab.se/In-english/" target="_blank">Stokab</a> project. Google fiber, while attractive, isn't going to be a real option for 99% of the nation. It will show what true broadband access is capable of (which the rest of the world is learning much faster than the US), but Google will never become a FTTH ISP. Local bodies can provide an alternative, and we think The Public Option via municipal broadband is the way to go.</div>
Vishal Misrahttp://www.blogger.com/profile/04748385949397825305noreply@blogger.com0