June 8, 2018

R Code

Glue is great for creating long strings of sql code. For example, imagine tables are stored in individual year-month files. Here is how we can use R to access these tables with a single query instead of writing out by hand every year-month combination.

library(tidyverse)
## ── Attaching packages ────────────────────────────────── tidyverse 1.2.1 ──
## ✔ ggplot2 3.0.0.9000     ✔ purrr   0.2.4     
## ✔ tibble  1.4.2          ✔ dplyr   0.7.4     
## ✔ tidyr   0.7.2          ✔ stringr 1.3.0     
## ✔ readr   1.1.1          ✔ forcats 0.2.0
## ── Conflicts ───────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
# create vector of year-month combos
yearmon <- expand.grid(year = 15:19, month = 1:12) %>% 
    mutate(month = str_pad(month, width = 2, pad = "0")) %>% 
    mutate(ym = paste0(year,month)) %>% 
    arrange(ym)

yearmon <- yearmon$ym

# glue together statements from each one
query <- glue::glue("sel id, date, value 
                    from table_{yearmon}")

query <- glue::collapse(query, sep = "\n UNION ALL \n", last = "")

Reading

A Feast for Crows and A World of Ice and Fire

September, 2018

Starting new job at Bank of America in consumer analytics

August 7, 2018

Just published an updated Weather Compare Shiny app. Things I learned:

I also found this cool reference for tools to spice up your shiny apps.

April 9, 2018

Reticulate for R-Python workflow in R Studio!

Reading

Onto the final book of LOTR

April 5, 2018

See Tim Roughgarden’s Coursera course on Algorithms for more.

The Master Method is a general tool for analysis of the run time of divide and conquer algorithms. The general form of the master method is:

\[ T(n) \leq a T(n/b) + O(n^d) \]

Here the parameters are:

The relationship between these three parameters result in three different run times:

Mergesort, for example, has the form of \(T(n)=2 T(n/2)+O(n^1)\), which gives us a run time of \(O(n \space log \space n)\).

April 2, 2018

Conditional expectation tells us that

\[ E(Y \vert X=x) = \sum_{i=1}^k y_i \text{Pr}(Y=y_i \vert X = x) \]

In words, the expected value of Y given a particular value of X is equal to the sum of the probabilities of each value of Y when X=x. Let’s say X takes two values, 2 and 3, and Y takes on many values from 0-100. For conditional expectation, we want to know what the expected value of Y when X=2. To do this we would just sum all the values of y multiplied by their probabilities whenever X=2.

The Law of Iterated Expectations tells us that

\[ E[Y] = \sum_{i=1}^l E(Y \vert X = x) Pr(x=x_i) \]

\[ E[Y] = E[E[Y \vert X]] \]

Essentially, the law of iterated expectations asks, what if we took a conditional probability of Y on X but then did it for every value of X. If you do this, you are essentially just taking the expectation of Y because you are going over the entire Y space.

Covariance can be written as:

\[ \sigma_{X,Y} = \sum_{i=1}^k \sum_{j=1}^l (x_j - \mu_X)(y_i - \mu_Y) Pr(X=x_j,Y=y_i) \]

March 27, 2018

New Constructs was taught in BAV for the first time with great success! Very engaging discussion and a great way to close out BAV.

March 13, 2018

I downloaded exercise files from Lynda for a course in efficient python programming. I wanted to preserve the original files, so I made two copies of the files in different folders. The problem is that I needed to change the name of all the files en masse to distinguish them from the original files. I modified some code I found online and used the following in the command line:

for file in *.ipynb; do echo mv "$file" "${file/begin./kt.}"; done

The files were named xx_begin.pynb. The above code renames all of these files by replacing the “begin” part with “kt”. A few points

March 5, 2018

Economist Why Oil Price is so High

Hedge Fund Delusion that Grips Pension Fund Managers

Edge Computing

Reading

LOTR Just finished the first half of the Two Towers.

February 28, 2018

Finished Airgas case C-D; still waiting for approval

February 26, 2018

An article that represents many months of my life cleaning and managing data: Are Buybacks Really Shortchanging Investment?

February 12, 2018

The case on New Constructs is submitted!

January 20, 2017

Check out the Weather Comparison App! Still some formatting issues, bits of missing data, and need for better documentation.

January 15, 2017

Weather Data is a R library for importing weather data from the NOAA. Will be interesting to see how detailed and expansive the data is. Goal is to build a shiny app that facilitates weather comparisons between cities.

January 14, 2017

The Vikings won in miraculous fashion. When Diggs took off down the sideline, I initially thought he was an idiot and was going to get tackled with no time left. But then he kept going and the rest is history.

January 9, 2017

Plotly has a very cool webiste for demoing there dash platform for interactive graphics.

Reading

Hobbit: Decided to have a complete picture so I need to read the Hobbit first. Interesting to note how much Tolkien developed the world the next 20 years after publication. The Elvenking is not even given a name.

January 6, 2017

infer is a tidy way of doing inferential statistics. Still early but, with modelr, would be very efficient.

January 5, 2017

I have been binge listening to the podcast ‘80s All Over’ while shoveling snow. It is a month by month recounting of the theatrical releases in US for the 1980s. A lot of interesting movies that seemed to have been lost to time.

Cold is incredible. Running low on oil.

January 4, 2017

Reading

Left Hand of Darkness: Just finished this “classic” that won both the Hugo and Nebula awards. I found the telling somewhat and the prose unimpressive. The gender politics was somehow both challenging and oddly traditional. Despite being told repeatedly that the characters are all androgynous black people, the characters are written like flabby old white men.

Lord of the Rings: Beginning to reread LOTR after my first read through in 2002. I began by reading the Prologue and the Appendices. It’s amazing that almost every detail of the making of the books and the movies has been documented so thoroughly. Also, apparently Gandalf is pronounced Gandalv??

Big snow day here in Watertown. Received 18 inches of snow. Sharp cold coming in soon.

December 6, 2017

The Orthagonality Thesis from Nick Bostrom supposes that any level of intelligence is capable of having any goal, i.e., super-intelligence does not mean super morality. An example is a super-intelligent paperclip maker that resolves to make the largest number of paperclips possible (and destroying the universe in the process).

December 5, 2017

Research Project idea: This analysis from Brookings and this are very similar in showing the value add of certain universities. The process is relatively simple: collect data, run regression, see difference between median salary actually observed and what the regression expects, then rank into percentiles. Could expand into a shiny dashboard.

December 4, 2017

Marriage in China: some families are paying years of salary in the form of bridal payments. Also, Germany saves 10% of pay but Chinese households save 38%!

Data Science Design Manual: I just started reading this book and it seems like a pretty great foundation for data science.

This page shows how to use acf and pacf plots to identify the numbers of AR/MA terms in an ARIMA model.

R for Data Science is a nice little book by Hadley on using R for foundational data science work. Not a math/stats book, just a book on R.

December 3, 2017

modelr is an early attempt at trying to apply elegant pipes to model building.

Idea: Tangled Hierarchies of Buddhist Literature: Buddhist literature as self-aware; reading itself when we’re away?

November 27, 2017

What the Buddha Thought by Richard Gombrich: Argues in the opening chapters that karma is the great leap of faith you have to take to understand Buddhism and cannot be ignored.

November 25, 2017

British Discovery of Buddhism: This book concerns the early British mappings of “Buddhism.” I think this is a critical fact that most have a hard time accepting: there was singular idea of Buddhism as a pan-Asian civilizational religion that was delivered wholesale to Europeans. It was a idea that was crafted over decades of discovery and interaction. The implication being that practitioners of Buddhism probably had wildly divergent ideas of what Buddhism was that what western scholars had crafted.

November 22, 2017

Ascent of Money: Relistening to the Ascent of Money by Niall Ferguson. The two most interesting narratives from the book is the rise of the Rothschilds. The Rothschilds were textile traders in continental Europe who opened up shop in England before the Napoleonic era. To continue their business under Napoleon’s protectionist continental system, the Rothschilds became adept at smuggling gold into Europe. Needing to fund their armies in Portugal, Britain turned to the Rothschilds to delivered gold.

November 16, 2017

Stata 15: new version of Stata includes dynamic documentations. This is nice but woefully underpowered compared to R Markdown.

The tidyverse package reprex allows for quick creation of reproducible examples when asking for help

Structural Equation Modeling allows for measuring dependencies between independent variables in regression. Much of this field was made possible by Stata releasing easy to use SEM.

November 3, 2017

The Storm Before the Storm: An interesting story about Marius: to demonstrate his vivacity despite his age, Marius exercised strenuously on the Campus Martius. Everyone laughed at him. Not to long after this, Marius had to flee Sulla on foot through swamps and over water, so his survival was due in large part to this exercise. While fleeing, Marius told the few people brave enough to accompany him the story of how he discovered a nest of seven eagles as a child — thus prophesying his eventual return to Rome to claim his seventh consulship

October 24, 2017

Millennium: Until 1434, Europeans thought that Cape Bojador along the coast of the Sahara was impassable. By 1522, they had completed a circumnavigation of the world.

Economist, Oct 21, 2017

October 20, 2017

Aberfan Disaster: Over 160 were killed in England in 1966 when a pile of coal spilled onto a village.

October 19, 2017

Millennium: The Black Death came to Norway via a ghost ship; The Black Death killed 45% of the England in 7 months over 1348—an annualized mortality rate of 77%; a low estimate is that 1/3 of Europe died during the initial phases of the plague.

Speaking of the end of the world, here is a list of near misses: list of air bursts

Chain of Thought - Power Law: Power Law -> Pareto Principle -> Fat Tailed Distribution

October 17, 2017

Millennium: Pious Folk Count of Anjou not really pious forced to build monasteries and go to Jerusalem

October 5, 2017

Would like to host R Markdown Notebooks and Jupyter Notebooks separately from the main blog so that the interactive bits can be rendered properly. I should be able to replicate this and this. The only trick will be getting it to work with the Pelican framework.

Economist, October 2

October 4, 2017

Napoleon: A Life: I’ve heard that the Siege of Petersburg in the American Civl War was first modern battle due to its extensive use of artillery and trench warfare. However, the Battle of Eylau during the Napoleonic wars may be an even earlier example due to its use of artillery and overall brutality. The French Marshall Ney commented, “What a massacre! And without a result.”

September 28, 2017

When interpreting regression results it is valuable to remember that the ceteris peribus interpretation is due to the coefficients being partial derivatives. A derivative is simply defined as

\[ \frac{\Delta y}{\Delta x} = \frac{f(x_0 + \Delta x) - f(x_0)}{\Delta x} \]

In words, if \(y=f(x)\) the change in y (\(\Delta y\)) per change in x (\(\Delta x\)) is how much how much a function changes (\(f(x_0 + \Delta x) - f(x_0)\)) given a change in x (\(\Delta x\)). Hence we get the standard derivative notation:

\[ \frac{dy}{dx} \equiv f'(x) \equiv \lim_{\Delta x \rightarrow 0} \frac{\Delta y}{\Delta x} \]

In both simple and multiple regression, we interpret \(\beta\)s in similar ways: a unit change in \(x\) results in a \(\beta\) amount of change in \(y\). But how can we ignore other coefficients in multiple regression? Because they are partial derivatives where one variable is held constant while the other is allowed to vary. For example, the derivative of \(f(x,y)\) while holding \(x\) constant can be written as

\[ f_x'(y) = \frac{\partial f(x,y)}{\partial x} \]

Napoleon: A Life: punishment should be infrequent but severe; “truth is so precious she should be surrounded by a bodyguard of lies”

Philosopher’s Toolkit: Richard Dawkins’ “selfish gene” is an example of a failed intuition pump—doesn’t mean at all what people think it does (he is referring to how genes optimize locally instead of globally for the organism)

September 22, 2017

Napoleon: A Life: notorious micromanager; emo youth—wrote excessively Romantic letters on suicide and his 1st wife Josephine; lived frugally when younger—only ate once a day; more concerned by mulberry nursery than revolution; “We have them now”

Petrozavodsk Phenomenon: an unusual and unexplained atmospheric phenomenon. Never heard of this until recently

Pythagorean Expectation: way of computing expected wins. Pro football uses exponent of 2.67, Redditor says this doesn’t make sense due to VC Dimension issues. Some more on VC dimensions.

IgNobel Award Winner: Elisabeth Oberzaucher and Karl Grammer, for trying to use mathematical techniques to determine whether and how Moulay Ismael the Bloodthirsty, the Sharifian Emperor of Morocco, managed, during the years from 1697 through 1727, to father 888 children

September 21, 2017

L1 norm for accuracy—less bias; L2 norm gives parsimony—less variance <|||> New favorite military title: 1st Sea Lord <|||> Ada Lovelace thought of using punch cards after being inspired by weaving loom <|||> 37% optimal stopping problems and Secretary problem

Jeremy Bentham referred to natural rights as “nonsense on stilts”—similar sentiment from other Enlightenment thinkers <|||> John Adams thought the idea that “all mean are created equal” was absurd

Strassen’s Subcubic Matrix Multiplication Algorithm: a divide-conquer-combine strategy that is completely non-obvious but hugely important because it saves on recursive calls.

July 7, 2017

Having a newborn means you can’t sleep but you’re too tired to work. So what do you do? Bing watch:

July 6, 2017

Super Valuation of Amazon: All you need to know about Amazon’s current price is this: “Never before has a company been worth so much for so long while making so little money: 92% of its value is due to profits expected after 2020.” For Amazon to justify it’s current price, it would have to become the most profitable company of all time. Is it possible?

The biggest hurdle for Amazon will be regulatory not economic. As the Economist continually points out, current regulations regarding monopolies and the like are outdated for times when custom data and smart algorithms can competitively set prices that normally would only appear under monopolies. Eventually regulations will catch up and Amazon as we know it might become multiple Amazon-lets.

Vix is the implied premium of insuring options against volatility in the markets. Over the past few months, Vix has been surprisingly low—and some fear it is the calm before the storm. This complacency prior to disaster seems to quantify an old trend of saying “the business cycle has been tamed—no more recessions” right before a financial crisis. Speaking of which…

Cape Fear: Right before the financial crisis of 2007, Ben Bernanke assured everyone that the looming housing crisis would be contained. We know how that ended. Ten years on and we’ve now just reached the same CAPE (cyclically adjusted price to earnings) ratio that we had in May 2007. Does this mean that equities are valued to high? The Shiller 10-year PE suggests that were about 10 percentage points above the historically observed median. However, if realized returns are a bad predictor of future returns, then how much should we trust these signals of another recession?

Akzovism: Shades of Airgas with an somewhat beleaguered industrial company rebuffing multiple takeovers and then being assailed by activists. The interesting turn here is the unique corporate governance structure wherein a foundation appoints board members thus negating attempts to vote in new slate of sympathetic directors.

Toshiba Delisting: Seemingly every month another titanic Japanese company begins to die off. Toshiba is under threat of delisting, Sharp is in trouble, and Takata just sold itself. But does this mean that Shinzo Abe’s economic reforms are failing, or is the greater accountability a sign of its success?

July 5, 2017

I recently wrote some code that can take a Stata dataset and produce a table with means and number of observations by variable and time series. The code is essentially agnostic to the data so long as all variables are wanted in the table and are numeric.

In early June, Apple held its annual developer conference were it announced a new lineup of computers. The price drop in MacBooks makes the Pro more reasonable, but the lack of 32gb RAM is unforgivable (some of us like loading huge datasets in R!). The touchbar is pretty cool, but I am not willing to pay the premium that it added to the price when it was first announced in 2016. The new iMac Pro, on the other hand, looks ridiculous, but I’m not really sure who it is for outside of Premiere Pro users.

July 4, 2017

A newish feature to RStudio are addins. These addins add a bit of convenient functionality to RStudio. For example, the base examples include an addin that can easily prettify R code.

A few other useful addins I have found:

If you have devtools installed in R then you can run the following code to install all of the above packages:

install_github("rstudio/addinexamples", type = "source")
install_github("BAAQMD/copydat")
install.packages("radiant", repos = "https://radiant-rstats.github.io/minicran/", type = "binary")
install.packages('ggedit')
install_github("lorenzwalthert/strcode")
  install_github("seasmith/AlignAssign")
install.packages("citr")

July 3, 2017

Any English speaking society that would think a name like “Praise-God Barebone” is a good name, is probably a little strange. The most innovative aspect of the Puritans (of which Praise-God was a member) was their ability to be strange and old-fashioned at the same time (after all, anyone can be new and strange). The Puritans were obsessed with keeping things the way they were. As David Hackett Fischer points out in Albion’s Seed,

“In the early records of the Bay Colony, the adjectives ‘new’ and ‘novel were pejorative terms.’ In 1639, for example, a special ‘day of humiliation’ was called in Massachusetts on account of ‘novelties, oppression, atheism, exesse, superfluity, idleness, contempt of authority…’”

The first thing most readers would zero-in on is the ‘day of humiliation.’ But Fischer focuses on something else: he notes that novelty is first in the list—even ahead of atheism. It seems odd to us in our ever-innovating, fast-paced society that even the most conservative person would prefer no change at all. Fischer suggests that part of their desire to keep things the way they were (are?) was to preserve a memory of their lost home in England. You have to remember that the Puritans were upper-middle class merchants with deep roots in East Anglia who were forced to flee Europe after being branded as ‘fanatics.’ This could also explain why colonial Virginians (a slightly less fanatical population) also considered words like ‘innovation,’ ‘novelty,’ and ‘modern’ to be pejorative terms. Ironically they to were forced to flee to the new world after the Puritans briefly took control of England under Oliver Cromwell.

You might be thinking, wouldn’t a strange name like “Praise-God” be novel? And you would be right—I don’t think anyone who wasn’t a Puritan ever had that name. The Puritans, however, could justify this “new” name because of how it was derived. A common naming convention for the Puritans was to simply open the Bible at random and point to a word. So in a sense there is nothing new at all about—God chose the name, at random, from the book he wrote several hundred years ago.

On a broader note, I wonder how much of the progression of our knowledge of earth’s history has also impacted our view of innovation. Until very recently (the 19th c. in fact), the idea that a species could go extinct was patently absurd. The world is as it was when God created it. If the world is (as we know now) constantly in flux, then it is no great leap to embrace the change we inevitably must experience.

July 2, 2017

An ancient Greek adage states: “a fox knows many things, but a hedgehog knows one important thing.” With no additional context, the adage simply bifurcates styles of epistemology without ascribing an upper-hand to either condition. Recently, however, I’ve seen the phrase used to describe winners and losers in the race for better, data-based predictions.

Philip Tetlock has centered his research on assessing and improving forecasts and makes clear his preference for foxes. His book Superforecasters, for example, details everyday people who consistently make better predictions than most experts regarding a wide range of world events. One reason for their success is their nearly obsessive collection of eclectic knowledge, making them “supernewsjunkies”. The core idea being that a critical quantity of information quanta can reveal patterns not recognized by large, over-arching narratives.

If you were to predict the future of oil prices, for example, would an intimate knowledge of OPEC’s history be all you need for accuracy? Or would you need to know how China’s aging population might affect aggregate demand for oil? Or what the median waterline on shipping vessels across the Pacific might reveal about global oil usage? What about CFOs who successfully hedge shale companies, keeping them afloat during a glut of oil? Many small details might give you an insight missed by the standard narrative.

Tetlock is not alone. Nate Silver in his book The Signal and the Noise also espouses a fox-outlook. His noted website 538 subtly alludes to this with its fox logo.

July 1, 2017

I have no data yet. It is a capital mistake to theorise before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.

—Arthur Conan Doyle

March 15, 2017

I love Rocksmith. This week I’m playing some E♭ standard tuning songs:

March 14, 2017

From the February 25th Edition

The Sanctity of Trade Statistics: A proposal to remove re-exports from official figures would inflate the trade deficit and bolster the current administrations case for tougher trade relations. The Economist comments, “the new administration seems prone to using statistics as a drunk uses a lamppost – for support rather than illumination.”

From the March 4th Edition

China and Currency Manipulation China has had its reputation as a currency manipulator longer than it has actually been a currency manipulator. This article, using their own measurements inspired by official US determinants, scores Taiwan and South Korea as greater depressors of their currencies than China. In fact, China has taken steps to strengthen the yuan to keep capital from flowing out.

March 12, 2017

KDNuggets runs an annual survey asking “What software you used for Analytics, Data Mining, Data Science, Machine Learning projects in the past 12 months?” The 2016 results are:

  1. R
  2. Python
  3. SQL
  4. Excel
  5. RapidMiner
  6. Hadoop
  7. Spark
  8. Tableau
  9. KNIME
  10. scikit-learn

A few observations:

March 11, 2017

Here There Everywhere: Quantum Computing: Just the other day I was wondering what it was like to be a US civilian just prior to the atomic bomb drop in 1945. Was there any idea that the US was working on a super weapon? What was it like to realize that your country had been secretly coordinating with the greatest minds on earth and spending billions of dollars developing an unstoppable instrument of destruction? What would be a current/future analog to this? Perhaps Russia attempts a major cyber attack on the US only to find out that America has developed a highly advanced quantum computer that not only renders the truly important parts of the US cyber infrastructure invulnerable, but also makes Russia an open book.

Timely enough, the technology quarterly for this week’s Economist focuses on quantum computers. The major take away is that progress in this field is rapidly advancing thanks to the growing and inevitable commercialization of quantum techniques. Quantum computers will soon provide more accurate location than GPS, more precise sensing abilities than anything available, unbreakable cryptography, and the ability to quickly solve problems in finance and engineering.

This last point reminds of the time saved in solving liner equations that came with the advance of analytical engines and early computers. Problems with 6+ unknowns would take weeks to solve, but advances in simple computers reduced the time to mere hours (your computer could probably do it in seconds).

From the March 11th Edition

March 10, 2017

Reading Walter Isaacson’s Innovators, I was struck by his telling of the first-long distance phone call. In the 1910s, AT&T was facing the expiration of their patent on the basic technology underlying telephone calls. To compensate, AT&T aggressively pursued new technologies such as long-distance phone calling. Sure enough, by 1915 Alexander Graham Bell recreated his famous first phone call with Thomas A. Watson, except this time a continent separated them and not a room.

I find it compelling how different it is today: namely, that a claim on intellectual property would be allowed to expire. Mickey Mouse, for example, should have gone into the public domain in 1984. However, due to intense lobbying, Disney has been able force changes in the law so that Mickey Mouse will almost always remain in the Disney vault. What kind of innovation has Disney denied the world by not letting their rodent mascot into the public domain?

March 9, 2017

On April 15, 1912 the RMS Titanic sank in the northern Atlantic taking the lives of approximately 1500 people. Stop anyone on the streets, and they certainly could tell you the general outlines of the disaster. But what about the sinking of the Sultana? On April 27, 1865 (April is the cruelest month), the steamboat Sultana caught fire and sank on the Mississippi river killing roughly 1700 – 200 more than the Titanic.

So why does everyone remember the Titanic and not the Sultana. One reason is timing: the Sultana sank the day after John Wilkes Booth was killed after himself assassinating President Lincoln. Another reason would be media coverage and popular interest: the Titanic catered to social elites who the media naturally gravitates towards. Add in the morality play of the ship “that even God could not sink” sinking and killing most on board, and you have the perfect storm of everlasting infamy.

Interestingly enough, this isn’t the only dyad of famous/forgotten maritime disasters. Everyone knows about the Spanish Armada – the most powerful naval force in the world dashed to pieces by the fury of a north Atlantic storm as it tried to attack England in 1588. Ultimately the loses Spain totaled 35 ships and 20,000 dead. But what about the English Armada? In the ensuing year, the English attempted a punitive expedition against Spain but met a similar fate losing 40 ships and upwards of 15,000 men.

A similar-in-magnitude pairing of remembered-and-forgotten can be found in the disastrous “kamikaze” invasion of the Mongols into Japan in the 13th century and the massive losses suffered by Roman naval forces in the second Punic war – both with high-end estimate losses of 100,000 men.

March 8, 2017

Reasoning will never make a Man correct an ill Opinion, which by Reasoning he never acquired

—Jonathan Swift

March 7, 2017

March 6, 2017

March 5, 2017

Joe Dimaggio’s parents were classified as enemy aliens due to their immigration from Italy (a Axis power during WWII) and forbidden from traveling farther than a five miles from their home. It should also be noted that Dimaggio’s own Italian heritage was controversial when he started playing in the major leagues.

March 4, 2017

From the February 11, 2017 edition

March 3, 2017

From the February 11, 2017 edition of the Technology Quarterly

March 2, 2017

Here are few interesting factoids about names of places in the ancient world:

March 1, 2017

I’m saying that at Leipzig all goods—silk, coins, shares in mines—lose their hard dull gross forms and liquefy, and give up their true nature, as ores in an alchemist’s furnace sweat mercury—and all mercury is mercury and can be freely swapped for mercury of like weight—indeed cannot be distinguished from it.

—Neal Stephenson. “Quicksilver.”

February 7, 2017

Listening

Reading

February 6, 2017

Super Bowl

Stunning comeback or stunning collapse? An excellent article details how Falcons bumbled and the Patriots exploited a 99% chance of victory. Despite the Falcon’s mind-boggling calls down the stretch (attempting passes despite Devonta Freeman averaging 8 yards a carry and only needing to run out the clock), the story during the course of the game was how little the Falcons offense was on the field.

I would also be remiss if I didn’t point out that the Patriots almost Seahawked themselves by throwing a nearly intercepted pass from the two yard line at the end of the game.

’Member when…

Given Tom Brady’s legendary success in the NFL, much talk has been given to him as the “Greatest of All Time.” This is a fine argument, but two caveats should be noted. Is he the greatest player of all time? That is a definitive no. Red Grange, a charter member of both the college and professional football hall of fame, is undeniably the greatest of all time. Is he the greatest QB of all time? Perhaps. Outside of Peyton Manning (who might be the best QB of all time), the biggest competition Brady has is Otto Graham: in his 10 year career he went to 10 championship games and won 7 of them.

February 5, 2017

February 4, 2017

From the February 4, 2017 edition

February 2, 2017

Most stars are binary in nature, i.e., most stars have a counterpart star that orbits a mutual barycenter. So what about our sun? What if our sun had a sibling? And what if that sibling was the cause of the mass extinctions that have periodically wracked earth? Such is the speculation that gave rise to the theory of Nemesis, a proposed star, small and very dark, that orbits with our Sun and periodically smashes through the Oort cloud every 26 million years, sending armageddon levels of debris to earth. In theory, this could include the impactor that lead to the extinction of the dinosaurs (if indeed they did go extinct from the impact of an extraterrestrial object).

This theory is quite engaging: it plays on our wonder at the enormity of the cosmos, the interconnectedness of time, as well as the possibility of mystery left in the universe. There is also something of the operatic in the idea of a long lost, evil twin who periodically ruins your life.

Unfortunately, as cool as this theory sounds, there is little evidence to support it. Despite being small and dark, such a body would have probably been detected by now. Furthermore the periodicity of mass extinctions has been challenged.

February 1, 2017

A firm rule must be imposed upon our nation before it destroys itself. The United States needs some theology and geometry, some taste and decency. I suspect that we are teetering on the edge of the abyss – John Kennedy Toole, A Confederacy of Dunces

January 28, 2017

“Languages build up to reflect specializations in a way of life. Each specialization may be recognized by its words, by its assumptions and sentence structures. Look for stoppages. Specializations represent places where life is being stopped, where the movement is dammed up and frozen.” – Frank Herbert, Children of Dune

January 27, 2017

Things that describe Kary Mullis:

One of these is not like the other…

The work of Lenoir, NC-born Kary Mullis has had an enduring, epoch-defining influence on biology in spite of his unusual predilections and controversial views,. In particular, his Nobel-prize winning work on polymerase chain reaction (PCR) was “highly original and significant, virtually dividing biology into the two epochs of before P.C.R. and after P.C.R.”

PCR is essentially a “kind of genetic photocopying, and it became the basis for all subsequent genetic science, from academic studies to police forensic work.” PCR relies on breaking done DNA and assembling a new copy of the strand with the help of heat-resistant polymerase. The end result of repeated applications are multiple copies of the original DNA strand. You can immediately see how this could be useful in forensic science: just a single hair fiber is now more than enough to produce sufficient DNA to test for comparisons to suspects.

As for Kary Mullis, well… maybe his autobiography is the best source of information: Dancing Naked in the Mindfield.

January 26, 2017

We all know Charlemagne: King of the Franks, First Holy Roman Emperor, coronated on Christmas Day in 800 CE. But what about Karl der Grosse? Or Carolus Magnus? This isn’t a mere translation issue — the equivalency of these two names points towards the fundamental similitude of Germany and France as well as the nearly universal appearance of the name “Charles.”

So Germany and France hate each other (see World War II, World War I, Franco-Prussian War… you know what — just check out the wikipedia page on the French-German Enmity). The EU was essentially founded to prevent another World War between the two — they even gave the Nobel Prize to themselves for so bravely refraining from war for a few decades. With this in mind, it’s a little awkward that France and Germany are historically the same. This issue is only possible because, until 1871, Germany wasn’t even a country and was more a general cultural idea.

Historically, France is a part of this idea — the Franks (France and Frankfurt, Germany) were a collection of Germanic tribes occupying the continental Europe north of the Alps. The most famous of the Franks (apologies to Pepin) is known almost exclusively by his French name Charlemagne (Char le Magne) despite that his name was actually Karl (Karl der Grosse). Add this to to wiki on “French-German Enmity.”

I remember thinking that it was pretty neat when I finally realized Charlemagne’s name is quite obviously “Charles the Great.” Investigating the relation between Charles, Char, and Karl, you quickly find an expansive list of other Karl derivatives: Carolina (North Carolina is named after Charles I, King of England), Charlotte (the city is named after the queen consort of George III), Caroline, Carla, Carl, Charlene, Carol, Carolyn, and Carlos.

January 25, 2017

R Markdown Notebooks is a new feature for RStudio that allows for independent and interactive chunk execution, quick rendering of LaTeX, and selective updating of html files (i.e., the entire page does not have to re-rendered each time). This is a very cool advancement in literate programming that takes the best features of iPython notebooks and applies it to a R development environment.

My continuing notes on the Introduction to Statistical Learning showoff some of these features.

\[ \begin{equation} \begin{split} E[(Y - \hat{Y})^2] &= E[(f(X) + \epsilon - \hat{f}(X))^2]\\ &= \underset{Reducible}{[f(X) - \hat{f}(X)]^2} + \underset{Irreducible}{Var(\epsilon)} \end{split} \end{equation} \]

I highly recommend you try it out. Reading about the features is quite underwhelming compared to seeing it in action.

To start a notebook you need to save a file as .Rmd and add a header like this:

---
title: "Notebook Title"
output:
  html_notebook:
    toc: true
    number_sections: true
  pdf_document: default
---

This will produce a PDF document with a table of contents and numbered sections.

January 24, 2017

From the January 28, 2017 edition

The Multinational Company: In Retreat: Multinationals, which account for a third of the world’s stock markets, are increasingly posting anemic performance results with declining profits and plunging ROE. The cause of this fall is that the opportunities that multinationals exploited in the international market are being “massaged” away: income is rising in the third world, mismatched tax treaties are being synchronized, and local firms are becoming more sophisticated. With a current rise in anti-globalism, the future direction and scale of multinationals are in question.

More articles:

January 23, 2017

’Member when…

January 22, 2017

Watching

Listening

Reading

January 21, 2017

When Lawrence understood, it was as if the math teacher had suddenly played the good part of Bach’s Fantasia and Fugue in G Minor on a pipe organ the size of the Spiral Nebula in Andromeda—the part where Uncle Johann dissects the architecture of the Universe in one merciless descending ever-mutating chord, as if his foot is thrusting through skidding layers of garbage until it finally strikes bedrock. – Neal Stephenson, Cryptonomicon

His outfit is a little bizarre. I thought he was a performer of some sort when I first came in, although I tried not to imagine the nature of his act. – John Kennedy Toole, A Confederacy of Dunces

January 20, 2017

In this classic story, the Chinese sage Zhuangzi (ca. 370 BCE) awakes from dreaming that he was a butterfly only to ask the question: am I a man dreaming of being a butterfly or am I a butterfly dreaming of being a man? Daoists, scholars, philosophers, poets, artists, and laymen alike have all been equally moved by so profound yet so simple of a question. Although the concept has been elaborated and explored numerous times (there are certainly more and less sophisticated takes on the idea), I believe Zhuangzi’s simple story has staying power due to its simultaneous depth and shallowness. The very form of the story, incredibly simple yet profound in implications, underlines the point that distinctions between deep and shallow — between anything in fact — is ultimately fruitless.

January 19, 2017

The earliest known representation of the Yin-Yang is not from China but from the late Roman Empire (ca. 430). The Notitia Dignitatum, which “details the administrative organization of the Eastern and Western Empires,” includes a depiction of the shield carrying a Yin-Yang symbol.

So does this mean China and Europe were trading ideas a millennium before the colonial period? Of course! Asia and Europe have been in contact for a long, long time. But Notitia is not the smoking gun. The current version of the Notitia is actually a Medieval copy of an ancient document — meaning that it’s about 1000 years younger than the 430 date.

January 18, 2017

The Australian Motoring Enthusiast Party is a micro political party that remarkably won federal representation in 2013. Let’s review a few spectacular facts from this Australian news article:

This is not an irrelevant incident: in 2016, “eight senators representing tiny parties… hold the balance of power [in the Australian Senate].” Nor is it isolated: in 2016, the Pirate Party won 10 seats in the Althing. Ultimately, this represents a potential pitfall of the Parlimentary system (a system that many Americans would like to try out given the current two-party deadlock in America.)

January 17, 2017

As someone who started with Stata and R for data analysis, I’ve always wanted to try a more general use language like Python. I worked through Python the Hard Way and played around with Dataquest (whose tips helped me build this blog). However, the best way of learning I have found is writing algorithms in Python.

Two of the cooler functions I have come across are zip and pop. zip essentially allows you to iterate over two iterables simultaneously. A trivial example would be

x = [1,2,3]
y = [4,5,6]

for i,j in zip(x,y):
  print(i,j)

the output would be [1,4],[2,5][3,6]. I can’t think of how much time this simple function would have saved me if it was available in Stata.

list.pop[i] is like the opposite of append: it removes the specified element from the list. This might sound simple, but it is by far the easiest way of removing something from a list that I have come across in R or Stata. My first use of pop came in writing the inversion counting algorithm. Here is a snippet (B and C are both arrays of numbers):

while B and C:
    if B[0] <= C[0]:
        outlist.append(B.pop(0))
    else:
        count += len(B)
        outlist.append(C.pop(0))

This chunk of code does the following:

  1. Checks if element 0 of B is less than or equal to element 0 of C
  2. If it is, then it removes that element from B and places it into outlist
  3. If it is not, then it increases count by the number of elements left in B (this is part of the counting inversions) and then removes that element from C and places it into outlist (the effect is two sorted arrays)

January 15, 2017

From the January 21, 2017 edition

Buttonwood: This weeks Buttonwood column addresses one of the pitfalls of the current trend towards protectionist policies: shielding established yet failing companies from displacement by global competition. Specifically, a paper from the OECD finds that companies that owe more than they make (interest-coverage ratio of less than 1 and are over 10 years older) might be stifling growth and lowering labor productivity. These older, less profitable companies use their lobbying clout (leveraging their employment size, no doubt) to pressure their respective governments into passing protectionist policies. To be sure, however, this isn’t a Right v Left issue: Sandernistas and Corbynistas alike were clamoring for anti-globalist policies as well.

Other articles:

January 14, 2017

NFL Conference Championship Games

College Basketball

’Member when…

January 12, 2017

Watching

Listening

Reading