Saturday, March 20, 2010
Don and Juan Killed My Marriage: Another look at divorce rates and same-sex marriage
“Gay marriage will lead to our divorce” is a line you might expect from someone against same-sex marriage.
In January of this year, Nate Silver conducted a preliminary analysis that seemed to show for the first time, that I know, that this argument was not only statistically untrue, but the opposite was true. He found that divorce rates fell from 2003-2008 in states with NO constitutional same-sex marriage bans enacted prior to 2008, while they rose in states with constitutional same-sex marriage bans enacted prior to 2008. He finished his post with the following "There is, however, probably now enough data on this subject to engage in more sophisticated longitudinal studies on this subject (more sophisticated than I have engaged in here), which might produce more robust conclusions."
Being a student who needs to do essays (otherwise he (I) fails out), I figured I would try to take Mr. Silver up on his offer. That is, I would try to add other variables and use a more "rigorous" (though not greatly) statistical technique to test the relationship between same-sex marriage bans and divorce rates.
What I found is going to be revealed in this post in three parts: 1. I am going to show what I believe to be serious errors in Mr. Silver's initial analysis. 2. I am going to complete my more "rigorous" analysis. 3. I am going to show why this whole thing is a crapshoot.
Before I begin, I want to say that I tried to reach out to Mr. Silver after finding what I did. Because of the massive emails he receives (he's a popular guy because of his many talents), I never received a response. I have sat on this information for over a month, but I finally had to complete this for school credit.
The Problems with Silver's Initial Analysis
I initially wanted to study to see if adding any more explanatory variables (changes in income, education, age, race, etc.) to Silver's numbers would change any of his findings. Before I did, I had to download the data as Mr. Silver instructed in his post. I gathered, in each state, the number of divorces in 2003 and 2008 from the Center of Disease Control (CDC) and the number of married people in 2003 and 2008 from the American Community Survey (run by the kind folks at the Census). Because some states are missing from the CDC data, we are left with 43 states and the District of Columbia.
Trying to be a legitimate academic (we will see if you agree), my first step was to reproduce Silver's changes in divorce rates (so that I could then add other variables). I quickly ran into a problem. Many of the rates I was finding did not match Mr. Silver's. I spent around twelve hours trying to figure out why, and then I got it.
According to Silver's post, he calculated the divorce rate in 2003 as such: (# of Divorces in 2003 / # of Married People in 2003) * 2 (every couple has 2 people). In 2008, he supposedly did the same calculation: (# of Divorces in 2008 / # of Married People in 2008) * 2. The change (as it will be for all calculations in this post) from 2003-2008: (rate in second period (08) / rate in first period (03)) - 1.
Except (perhaps because his post was at 4:12 am... pot calling kettle black), Mr. Silver did the following for calculating the divorce rate in 2003: (# of Divorces in 2003 / # of Married People in 2008) * 2. For calculating the divorce rate in 2008, he made the same error in reverse: (# of Divorces in 2008 / # of Married People in 2003) * 2.
The major problem with this error is that many of the states that have passed same-sex marriage bans are gaining population, while many of the states that have not passed bans (or legalized same-sex marriage in the case of Massachusetts) are losing population. In mathematical terms, Silver's mistake artificially increases the divorce rate in 2003 in states with no constitutional bans (because the # of married people in 2008 in these states is lower), while it artificially decreases the rate in states with constitutional bans (because the # of married people in 2008 in these states is higher). The exact opposite effect happens when calculating the 2008 divorce rates. Overall, this spuriously drops the change in divorce rate by a much larger margin than it should in states with no ban, while it spuriously raises the change in divorce rate in states with bans.
In addition to this problem, Mr. Silver's initial analysis did not include the District of Columbia (an area that just legalized same-sex marriage). The District saw a major increase in divorce rates from 2003-2008. It also seems that Mr. Silver mistakenly assigned a constitutional ban to the State of Washington.
What follows is a table with Mr. Silver's found divorce rate changes from 2003-2008 and my calculated (and verified by someone a lot smarter than I) divorce rate changes from 2003-2008. States in red had a constitutional ban against same-sex marriage implemented prior to 2008; states in black had no ban nor allowed same-sex marriage; and, states in blue legalized same-sex marriage prior to 2008.
As you can see the differences between the two datasets is pretty astounding. Some states changed their relative position in change of divorce rate by as many as 21 spots. The average change per state (or district) is 8.
The two states with the most negative divorce rate changes from 2003 to 2008 are now states that passed same-sex marriage bans prior to 2008, Alabama and South Carolina. Also, states with a ban had an average decrease in divorce rates of 2.9%, not an increase of 0.9%.
The area with the biggest increase in divorce rate is now Washington, D.C., which did not have a ban. Overall, areas that did not pass a ban actually saw a decrease of 6.2%, not 8.0% as Mr. Silver found. And while states (+ the District of Columbia) that did not pass constitutional bans against same-sex marriage had a larger decrease in divorce rates relative to those states that did pass a ban, the difference between the areas that had a ban and did not have a ban is NO longer statistically significant.
Still, I wanted to complete a slightly more sophisticated analysis.
A New Model
The analysis above is not as in-depth as I would like. The model does not take into account the fact that not all constitutional bans against same-sex marriage prior to 2008 went into effect in 2004. 8 states had bans that went into effect in 2006. How do we know these states did not see an increase (or decrease) in divorce rates from 2003-2006, only to see the exact opposite happen from 2006-2008? At the same time, three states had bans enacted prior to 2003 (Alaska, Nebraska, and Nevada). How do we know these states did not see an increase (or decrease) in divorce rates in the immediate aftermath of their bans passing, only to see the opposite happen in later years?
I also wanted to take Mr. Silver's advice and add some other explanatory variables that might better explain divorce rate. The following variables have been linked with divorce (excuse the colon after the verb): political affiliation, income, race, education level, and age. People living in blue states are less likely to get divorced as are richer people, whites, more educated, and older.
So what exactly did I do? Using the American Community Survey for marriage data and CDC for divorce data, I went all the back to 2001 and calculated the change in divorce rate for three two year periods (2001-2003, 2003-2005, and 2005-2007) in all states with available data + D.C. I also calculated the change in median household income, percentage whites make up of the population, percentage of those over 25 with a bachelor's degree, and median age. I measured political affiliation using the 2008 Cook Partisan Voting Index (PVI). I would have liked to have gotten a political affiliation variable that changed with the years, but unfortunately those figures are not available. As it is, PVI will act as a constant. Quibble statistically, if you must.
The three intervals allow us to take care of the problems described in paragraph 1 of this section. In each time period, we can separate out the states that had a ban in place in the first year in a given interval (e.g. Nebraska had a ban in both 2001 and 2003), had a ban take effect during a given interval (e.g. Nevada did not have a ban in 2001, but passed one in 2002), or did not have one during a given interval (e.g. Alabama's ban did not go into effect until 2006). Therefore, we can see the overall differences in divorce rates between those states that had the constitutional ban for more than a year (Nebraska in 2003), just passed a ban (Nevada in 2003), or did not have a ban (Alabama in 2003).
I did not use the 2004-2006 interval because 18 states had bans for part of one of these years, while not having one in the other part of the year. In addition, Massachusetts legalized same-sex marriage in the middle of 2004. Kansas and Texas had bans implemented in the middle of 2005, so I have eliminated their 2003-2005 and 2005-2007 data. The elimination of data in these instances is in an effort to make my groupings (ban vs. no ban vs. marriage) as "pure" as possible. Despite these eliminations, I still had 131 observations.
With these observations, I found the following results. Keep in mind this is preliminary (as is this post... hey it's a blog). I am more than happy to share my data with others, if they feel like they can do something better statistically. If you see a mistake, let me know in the comments.
First, a simple regression of just change in overall divorce rate against a state having a ban by the end of a interval: states that did not have a ban had on average a 3.5% drop in divorce rate over each two year interval, while states that did have a ban saw no change in divorce rates. This difference was statistically significant with 95% confidence.
Second, I broke down the ban groupings between a newly implemented ban and having a ban throughout a given interval. I ran simple regressions of divorce rate against a state having a newly implemented ban (withholding observations of those areas that had a ban at the beginning of interval) and against a state having a ban at the beginning of a period (withholding those areas with newly implemented bans). It turns out that only states with a newly implemented ban (rise of .05%) differ significantly with those that have no ban (drop of 3.5%). Those states that had the ban in the beginning of an interval actually had a drop of .05%. I will allow you to draw your conclusions from the Massachusetts numbers, but because it is only one observation I would ignore it. Overall, if you believe these numbers, it would mean that states that passed a ban initially vary with concern to divorce rate from those states that do not pass a ban; however, this difference begins to disintegrate with time.
Third, I added PVI plus the other explanatory variables (income, race, education level, and age) and their changes over the three intervals to the freshly implemented ban regression model. Even with the addition of these explanatory variables, divorce rates in newly implemented ban states were still significantly higher (with 90% confidence) than those states with no ban. No other variable was statistically significantly different, but states with a higher PVI (more Republican) were more likely to have higher divorce rates. This may suggest that the bans on same-sex marriage variable is a stand in for something happening in red states. Religion perhaps?
Fourth, I added these explanatory variables to the model where we measured the differences between states that had a ban for longer than a year vs. no ban. No variables were statistically significant, but states with same-sex marriage bans were still more likely to see higher divorce rates.
Overall, the addition of these other explanatory variables did not really do anything to help explain away what we saw in our "second" step.
As I admitted for the model in our first section, I will admit the models in our second section are elementary. All one has to do is look at what happens when we concentrate on only changes during the third interval (2005-2007).
In this third period, the divorce rate change among states without a ban rose by 0.7%, rose by 0.5% in states with a newly implemented ban, and dropped in states that had bans prior to 2005 by 0.5%. In other words, the relationship we were seeing when looking at the three intervals (01-03, 03-05, and 05-07) in combination disappears and somewhat reverses itself when only looking at 05-07.
When we add the explanatory variables, states with new bans (those passed in 2006) are still more likely to see a drop in divorce rates than those in states with no ban. The difference is not statistically significant, and none of the other explanatory variables have a statistically significant impact on predicting divorce rates.
HOWEVER, when we compare those states that had a ban prior to 2006 (excluding Kansas and Texas for the reason stated earlier) to those states without a ban, states without a ban are on the periphery of being statistically significantly (with 90% confidence) more likely to see a rise in divorce rates. Unlike the other models we ran, states with increases in income were also more likely to see drops in divorce rate. We would expect this as financial problems can pull families apart.
Does this mean that we are likely to see divorce rates continuing to rise in states that do not pass a ban relative to those states that have? I VERY HIGHLY DOUBT IT. The truth of the matter to quote a friend of mine is that most likely "no relationship exists".
I truly believe that people get divorced because the person they thought they were compatible with turned out not to be. Truthfully, if a woman I thought was tough, smart, and kind (and if you are those please send me an email... TY) turned out to be weak, not as smart as I hoped, and mean, then I would want to break it off. I think most of us do not have our relationships impacted if Laser and Blazer decide to get married. Then again, I really do not know.
What I have done here is pretty much show what we thought already. I have run a slightly more complicated model than Mr. Silver had previously implemented. I have shown that the relationship between same-sex marriage and divorce rates really does not exist in any consistent fashion. I would be more than happy to have other people suggest variables or ways to carry this out, but at the end of the day I do not think a relationship exists.
Subscribe to Posts [Atom]