How to Build a Website From Scratch

January 23rd, 2014

When I signed up to build the Open Data 500 website, I wanted to go through the entire process of making a website from scratch. Full stack. Just to sort of see what it was like.
After spending 5 entire 10-hour days trying to trouble shoot a feature on the site, I decided to write a post on the skills needed to build an entire website from scratch.

To build an entire website from scratch you need to know the following:

  • HTML5
  • CSS
  • JavaScript
  • jQuery
  • D3
  • ParsleyJS
  • Modernizr
  • Tornado
  • Python
  • MongoDB
  • Mongoengine
  • CSV
  • JSON
  • geoJSON
  • Regular Expressions
  • Seamless
  • Heroku
  • Command Line
  • Git / Github
  • Google Analytics
  • MailChimp
  • DNS Records (A, CNAME, MX, etc)
  • Oh yeah, go directly to hell, GoDaddy
  • Polar Vortex Survival Skills
  • Basic Pharmacology
  • UX
  • UI
  • FU
  • F712U
  • Scheme (might as well)
  • Creative Commons Licensing
  • PHP (throw in a couple more languages, just in case)
  • Java
  • Ruby
  • C#
  • C♭
  • Perl
  • .NET
  • Obviously not WordPress
  • Ballmer Peaks
  • Double-team keyboarding
  • Windows
  • Mac
  • Linux
  • Atari
  • SNES (you’re welcome)
  • Brainfuck
  • SSL
  • HTTP
  • API’s
  • SOAP
  • LDAP
  • TCP/IP
  • WOFF
  • DOM
  • Cookies
  • XSRF
  • RSS
  • XML

I think that’s about it. I’f you’re just beginning with web development. Good luck. You’re almost there.

26

(Seriously, though. Keep it up, the road is long and arduous, but it’s totally worth it)

 

 

My Social Network

October 13th, 2013

I was playing around with Gephi, and I loaded my Facebook data to visualize my social network (or at least my Facebook social network). This is the result (click for full size).

 SocialNetwork

As you can see the network is pretty modular, which is to be expected since I’ve lived in 6 cities. There are 13 communities:

  1. High School, mostly my graduating class (21.97%) – Green
  2. The rest of Monterrey (17.41%) – Red
  3. ITP (16.4%) – Acqua
  4. Model UN (14.23%) – Light Blue
  5. UT Austin (12.57%) – Fuchsia?
  6. Oklahoma City (5.71%) – Purple
  7. Family, extended family, and family friends (5.13%) – Dark purple
  8. Schlumberger (3.32%) – Lime Green
  9. GovLab (1.3%) – Yellow
  10. NYCDigital (0.79%) – Orange
  11. Students For Sensible Drug Policy (0.72%) – Dark Blue
  12. Las Chilangas de Nueva York (0.22%) – Dark Blue inside ITP blob
  13. The group of Canadians I randomly befriended on a bus one day. (0.22%) – Tiny Light Green Offshoot from large Green blob

I filtered out those nodes which had less than 2 degrees (less than 2 mutual friends), but it was interesting to see the lonely nodes on my network. Those are mostly people that I have encountered while traveling alone or have randomly met. The graph contains 1,384 nodes (friends) with 35,226 edges (connections) between them. The longest path (network diameter) between two of my friends (without going through me) is 8. The huge blue dot in the middle is Gaby, and she is connected to 7 of my 13 communities and shares friends. In second place is Chantel who knows everyone in Monterrey.

Making your own graph

If you want to do this for your own Facebook data, go to http://snacourse.com/getnet. Authorize the app. I selected all options in case I want to use that data later. Click on the ‘click here‘ link in Step 2. The app will need to scrape your Facebook and this might take a while if you have a large network.

You’ll also need to download Gephi, an open source visualization software.

Once you’ve downloaded your data and Gephi, open Gephi and File->Open your data file (default settings should be OK). You’ll see a bunch of dots arranged in a square in the middle of the screen.

Screen Shot 2013-10-13 at 12.04.02 PM

You’ll need to tell Gephi to reorganize the graph. On the bottom left you can choose a Layout. I chose ForceAtlas 2, checked Dissuade Hubs and Prevent Overlap, and set Gravity to 50.

Click Run. You’ll see the dots start to move around. Depending on the size of your network, it might take a while before you start seeing a discernible pattern. You can click on an individual node to find information about it by selecting the Edit tool in the toolbar (bottom-most tool). The node info will be displayed on the edit tab next to the Partition and Ranking tabs.

Screen Shot 2013-10-13 at 12.19.17 PM

If you want to remove the lone nodes and just show your one giant network, on the right of your screen you’ll see a Statistics and Filters tab. Click on Filter -> Topology, and drag “Giant Component” below to where it says ‘Drag filter here‘. Click Filter at the bottom. I also filtered out nodes with less than 2 degrees. Drag ‘Degree Range‘ into your Queries as well. When selected, you’ll see Degree Range Settings at the bottom. Drag the sliders or double-click the numbers to edit them. (Don’t click Filter again, the button works like an On/Off switch, and it was already on from the previous step).

Degrees

Before sizing the nodes by degree (in this case degrees represents mutual friends), let’s calculate the Average Degree. Under Statistics, click on Run next to Average Degree. You’ll get a result for average number of mutual friends across your network and you’ll get a nifty distribution graph. Usually this looks like a power-law distribution.

Now, go to the top left and click on the Ranking tab. In the drop down menu, select Degree. You can visualize with color, size, label color, or label size. I chose Size, but feel free to play around. Choose a range that fits best for your network, and hit Apply.

Screen Shot 2013-10-13 at 12.15.41 PM

By the way, if you graph isn’t changing much anymore, you can stop the ForceAtlas 2 Layout process. Click on Stop. The dots should stop moving.

Communities / Modularity

To color the different communities, you’ll need to calculate Modularity. It’s under the Statistics tab on the right. Click Run. Press OK for the default settings. Again, you’ll get a nifty distribution chart.

Go to the Partition tab on the top left. Under Nodes, click the Refresh Button Screen Shot 2013-10-13 at 12.27.49 PM. Select Modularity Class from the drop-down menu. If you don’t like the colors, you can right-click inside that window and select Randomize Colors. Or click on the individual colors and manually select your colors. Once you’re happy with the colors, click Apply.

Awesome! You’re own social network graph. Gephi is a lot of fun to play around, and I encourage you to do so. The Gephi website has a bunch of tutorials you can follow that will teach you some of the awesome things you can do. To save your graph as a PDF, click on Preview on the top-top left. Feel free to play around with the settings, they’re pretty straight forward. When you’re done, just click on Export SVG/PDF/PNG at the bottom left.

I’ll try to make my graph prettier. As soon as I can get Illustrator to open up this tiny file.

What One Database Marketing Company Knows About Me

September 8th, 2013

It’s no surprise that marketing companies gather data about you to sell off to advertisers who then deliver targeted ads via mail, email, or while you surf the internet. Sometimes it’s even creepy how much they know about you. So far, it’s been a bit of a mystery finding out exactly how much of your information these companies have. A few days ago one marketing technology company, Acxiom, launched a new service called AboutTheData.com which allows people to take a peek into how much information the company has gathered on them.  Acxiom is no small marketing company. According to the NYTimes, it has created the world’s largest commercial database on consumers. I decided to give the service a try to see just how much data this company had about me.

Since this is such a large company, and I’m such an active internet user, I expected to find Acxiom to have gathered a lot of information about me. I was slightly disappointed–or relieved–when I found out that they didn’t have that much information on me at all (honestly, I don’t know how I should feel about this). Before going into the data, here is a little more information about where this data comes from and what we are shown.

According to Acxiom, this data is collected from:

  • Government records, public records and publicly available data – like data from telephone directories, website directories and postings, property and assessor files, and government issued licenses
  • Data from surveys and questionnaires consumers fill out
  • General data from other commercial entities where consumers have received notice of how data about them will be used, and offered a choice about whether or not to allow those uses – like demographic data

The data they show us, is their “core data”. This data is used to to generate the modeled insights and analytics used for marketing, which they do not show. Acxiom says that we are shown all of their core data. They make no mention about whether there is other non-core, non modeled insights data.

The site allows you to view data from six categories categories. Below is the information that has been gathered on me. Economic and Shopping data is over the past 24 months.

Characteristic Data: Male, Hispanic, inferred single
Home Data: No data.
Vehicle Data: No data.
Economic Data: Regular credit card holder (as opposed to Preimum/Gold), Regular Visa, 2 cash purchases (includes checks), 1 Visa purchase.
Shopping Data: $139 spent on 3 purchases (the ones referred to above?), 2 offline totalling $100, average $50 each (one purchase < $50, the other >$50, so I guess it’s a coincidence they add up to $100), 1 online for $39. My supposed interests include books, magazine, Christmas gift purchase, ethnic products (??), lifestyles, interests, and passions.
Households Interests Data: No data.

It makes sense that there is not be a lot of information about my home data or vehicle data, since I currently own neither (although there was no info on my previous vehicle ownership). Perhaps car and homeowners would have these sections filled out entirely. The household interests category is meant to include data related to interests of me or people in my household (examples given from the site include: gardening, traveling, sports). Not so surprised this is also empty, but I’m not sure why they guess that my shopping interests include ethnic products and yet they are not able to guess that I enjoy traveling. As for Characteristic Data? My Twitter feed should be enough to reveal that I’m a single male hispanic. Since you have to provide your name, email, address, and last 4 digits of your SSN, it’s pretty safe to assume that they also have this information.

**To skip Luis’ short history of shopping, jump to the next paragraph.
Economic and Shopping Data provide a little more hints as to where the data are coming from. First of all, they only have three purchases. That’s it. Out of the 3,100 card/check purchases I’ve made over the past 24 months, they have 3. I tried looking for two offline purchases on my Mint which add up to $100, but this proved to be a very difficult exercise. Even after filtering offline purchases and sorting data, there were too many possible combinations. For now, those two offline purchases remain a mystery. I was able to find a suspect for the online payment of $39. The most suspicious purchase came from a $39 seat upgrade at United Airlines. I can’t be sure if this is the one since I happened to buy a $39 upgrade, plus a plane ticket which does not show up in my AboutTheData. However, my suspicion arises from the fact that Mint had prepared a targeted ad for me by placing a green flashy dollar sign next to the purchase. This also could’ve been a coincidence.

Conclusions/Best Guesses
Given the fact that I spend A LOT of time on the internet and the high amount of purchases I’ve made over the years (I should cut down on those), I am surprised that Acxiom does not have more data about me. Basically, they know I’m a single, male, hispanic, and that’s about it. I can’t possibly imagine what they could gather from the rest of my data that’s worth $$$ to advertisers. Additionally, it seems a lot of their data comes from publicly available government data sets (home and car ownership), and–at least in my case–not a lot of data comes from neither my online habits or my shopping habits. I presume most of my important data is owned by Facebook and Google, and I’m pretty confident that they do not sell/share my data with Acxiom.

Last thought: AboutTheData let’s you edit your data so that you can receive more accurate targeted advertising. I’m curious to know who uses Acxiom data to target me, so I would’ve loved to enter distinctive preferences that do not apply to me (yet) such as “pregnancy”, “colonoscopies”, “underwater basket weaving”, or “Cook Islands National Women’s Football League” to see where these ads pop up. Unfortunately, AboutTheData only lets you change the above mentioned interests to ‘true’ or ‘false’. I guess they thought about the trolls.

RT @MartinLutherKingJr I Have a Dream… #CivilRights

August 28th, 2013

On August 28, 1963, Martin Luther King Jr. delivered his famous ‘I Have a Dream’ speech during the March on Washington for Jobs and Freedom. The event, attended by over 250,000 people turned out to be a defining moment for the Civil Rights Movement. Fifty years later, the country has made great strides towards equality, but a lot remains to be done not just in terms of equality, but also in a wide range of social issues. While some of the issues of today are the same as the issues fifty years ago, new technologies and new ideas have quickly empowered us to react to these struggles in ways where we don’t yet fully understand its effect. The widespread adoption of the internet and social media have by no means replaced traditional marches like Dr. King’s, but rather it has augmented the form in which people participate in social movements. Going forward, it’s important to ask what is the effect of these new forms of activism. Will traditional marches like the one fifty years ago ever be replaced? Will online forms of protest ever match the effects of, for example, the 60′s anti-war movement? What would Dr. King make of the hundreds of thousands of people who tweet for a cause?

Screen Shot 2013-08-28 at 9.16.38 AM

 

Click on the image to see the Processing sketch. Code included.

*This post was posted in this blog in tandem with The GovLab.
**No… the tweets are not live.

Red Burns

August 24th, 2013

I’m pretty bad at words as it is, and in moments like these, I’m especially bad at words. So I usually don’t say anything, out of fear that whatever I say will sound stupid. So instead of mine, here are hers.

Red would present this on the first day of her Applications to the incoming ITP class. (Transcribed by Chris Selleck, posted to the ITP Alumni list by Michael Colombo)

 

What I want you to know:
That there is a difference between the mundane and the inspired.
That the biggest danger is not ignorance, but the illusion of knowledge
That any human organization must inevitably juggle internal contradictions – the imperatives of efficiency and the countervailing human trade-offs
That the inherent preferences in organizations are efficiency, clarity, certainty, and perfection.
That human beings are ambiguous, uncertain, and imperfect.
That how you balance and integrate these contradictory characteristics is difficult
That imagination, not calculation, is the “difference” that makes the difference
That there is constant juggling between the inherent contradictions of a management imperative of efficiency and the human reality of ambiguity and uncertainty
That you are a new kind of professional – comfortable with analytical and creative modes of learning
That there is a knowledge shift from static knowledge to a dynamic searching paradigm
That creativity is not the game preserve of artists, but an intrinsic feature of all human activity
That in any creative endeavor you will be discomfited and that is part of learning
That there is a difference between long term success and short term flash
That there is a complex connection between social and technological trends. It is virtually impossible to unravel except by hindsight.
That you ask yourself what you want and then you work backwards.
In order to problem solve and observe, you ought to know how to: analyze, probe, question, hypothesize, synthesize, select, measure, communicate, imagine, initiate, reason, create
That organizations are really systems of cooperative activities and their coordination requires something intangible and personal that is largely a matter of relationships
What I hope for you:
That you combine that edgy mixture of self-confidence and doubt
That you have enough self-confidence to try new things
That you have enough self doubt to question
That you think of technology as a verb- not a noun
It is subtle but important difference
That you remember the issues are usually not technical
That you create opportunities to improvise.
That you provoke it. That you expect it.
That you make visible what, without you, might never have been seen
That you communicate emotion
That you create images that might take a writer ten pages to write
That you observe, imagine and create
That you look for the question, not the solution
That you are not seduced by speed and power
That you don’t see the world as a market, but rather a place that people live in – you are designing for people – not machines
That you have a stake in magic and mystery and art
That sometimes we fall back on Rousseau and separate mind from body
That you understand the value of pictures, words, and critical thinking
That poetry drives you, not hardware
That you are willing to risk, make mistakes, and learn from failure
That you develop a practice founded in critical reflection
That you build a bridge between theory and practice
That you embrace the unexpected
That you value serendipity
That you reinvent and re-imagine
That you listen. That you ask questions.That you speculate and experiment
That you play. That you are spontaneous.That you collaborate.
That you welcome students form other parts of the world and understand we don’t live in a monolithic world
That each day is magic for you
That you turn your thinking upside down
That you make whole pieces out of disparate parts
That you find what makes the difference
That your curiosity knows no bounds
That you understand what looks easy is hard
That you imagine and re-imagine
That you develop a moral compass
That you welcome loners, cellists, and poets
That you are flexible. That you are open.
That you can laugh at yourself. That you are kind.
That you consider why natural phenomena seduce us
That you engage and have a wonderful time
That this will be 2 years for you to expand- take advantage of it
Appolinaire said: – Come to the edge, -It’s too high, – Come to the edge, – We might fall, – Come to the Edge, – And he pushed them and they flew

 
 

R.I.P. Red Burns

 

Pushing Policy Through Mexican Telenovelas

August 14th, 2013

In recent Mexican news, there has been some talk about an energy reform. One of the biggest and most contentious issues being discussed is whether the state-owned oil company, PEMEX, should be allowed to receive foreign investment. Whether it should or it shouldn’t is not the topic of this post.

A couple of months ago, a YouTube user uploaded this clip from the telenovela Corazón Indomable.

Translating, and attempting to keep the tone of the conversation, the dialog goes something like this:

Woman 1: “Is there an inconvenience ma’am?”

Woman 2: “I didn’t think that a foreigner could have property in this island.”

Woman 1: “Yes, we are happy that is so. We would be sad if we didn’t have investment from abroad.”

Woman 2: “Why?”

Woman 1: “Because places prosper with everyone’s talent, national and foreign.”

Woman 2: “Hmm, but foreigners take the money with them, no?

Woman 1: “Money comes and goes, what you’re talking about with the Emir, he practically leaves it here because he spends it here.”

[Some talk about the Emir.]

Woman 2: “Well, every day you learn something new.”

Woman 1: “And what did you learn today?”

Woman 2: “Well, that foreign investment is really necessary and convenient.”

Woman 1: “Don’t doubt it. Without disparaging the national ones, of course.”

Woman 2: “Well, thanks for everything and good day.”

Woman 1: “Good day, excuse me.”

This telenovela is produced by Mexico’s largest media company (and largest in the Spanish-speaking world), Televisa. Televisa generated a lot of controversy during last year’s presidential election because of an alleged secret collaboration between the PRI and Televisa. The media company has been criticized for it’s low quality content and often ridiculous telenovelas (if you don’t speak Spanish, I’m sorry, you’re missing out), but I’m not sure I have seen this blatant use of telenovelas to convince the public of a political agenda before. It even gets didactic in the end when the woman asks “And what did you learn today?”

In a country where only two companies (Televisa and TV Azteca) own practically all TV content in Spanish, it is becoming increasingly important to make sure more people have access to the internet, where they would have access to a broader variety of media content, and as the Soraya meme shows, would even have the ability to ridicule such horrible TV .

GovLab Post: Democratizing Policymaking Online: Liquid Feedback

June 13th, 2013

Note: This post was written for the GovLab on June 10. You can view this same post at The GovLab’s site here.

Liquid Feedback
This week, Beth Noveck kicked off her talk at the Personal Democracy Forum conference by reflecting on the the MoVimento 5 Stelle (M5S), Italy’s 5 Star Movement, and their use of Liquid Feedback (LQFB), a software for policymaking and political discussion. After its initial deployment by the German Pirate Party, the software has gained a lot of popularity over the past few years, and most recently it has been adopted by several M5S groups in regions such as  Lombardy, Lazio and Sicily. True to their principles of participatory democracy and free access to the internet and information (and in response to criticism about how they run their business), it is no surprise that these two parties have been searching for a platform to engage their members more directly. But is Liquid Feedback the answer?

Liquid Democracy
Before we get into Liquid Feedback, lets introduce the concept of “liquid democracy.”

Think about how you would vote on where to go to dinner when hanging out with five of your friends. The five of you would sit around the living room, discuss what each of you are in the mood for and then vote on a place that would suit most people’s cravings. This is a very simplified version of  a direct democracy. However, what if instead of 5 friends, you’re hanging out with 30 friends? You might not remember the last time you had to discuss and agree on where to eat with 30 people, and that would be because as a group grows larger, discussions get longer and reaching a consensus gets harder. Also consider that Billy and Jane are from out of town, and what do they know about local food?

Direct democracy — one person one vote — does not scale well. The voters might not always be knowledgeable on the matter being discussed. That is why today, many governments use a form of representative democracy, where people vote on representatives they trust who will represent them when voting on policy decisions.

As most voters can attest, however, your representatives may not have expertise on every topic and won’t always share your same opinions on every single issue. We don’t want Billy and Jane choosing the restaurant.

Liquid democracy tries to take the best of both direct and representative democracy by allowing the voter to decide whether to delegate her vote to a representative on a given issue or simply vote on her own. Say you’re an expert on education, wouldn’t it be great if you could have your representatives vote for you on all health care issues, but when it came down to education issues, you could cast your own vote? This is what liquid democracy attempts to do. This video, by German designer Jakob Jochmann, provides a great introduction to liquid democracy. Liquid Democracy would have been unworkable prior to the Internet but is becoming a reality today and ready for prime time testing.

Delegate your votes to someone else who in turn can give their votes to another person they trust.

Liquid Feedback – How it works
Liquid Feedback is an open-source software created to facilitate Liquid Democracy. It enables policy discussions and decision-making with this kind of proxy voting system. It allows participants to propose policy, revise anyone’s proposals or propose alternatives, and vote on issues themselves or by proxy (someone else vote’s for them).

Here’s how it works:

Any member can propose policy. For the proposal to be taken to a revision period, it needs to gather 10 percent quorum within a certain amount of time. Once in the revision period, any member can set up an alternative proposal and over the next few weeks members vote up or down on the available proposals until a winner emerges. The voting is where it gets interesting. A member can decide to vote individually on an issue, but it would be a daunting task to read and go through every policy paper available.

Liquid feedback allows you to give your vote to someone you trust would vote on your side of the issue. Additionally, the person you delegate your vote to can also give his vote, along with all of his votes, to someone else. Very quickly, people could gain a lot of voters and hence a lot of power, but the system allows members to reclaim their votes at any given time, so if someone wants to keep their voters, they need to keep constantly working for them. “We want effective people to be powerful and do their work, but we want [the grassroots] to be able to control them,” says Ingo Bormuth, the spokesman for the Berlin Pirate Party.

Liquid Feedback allows members to delegate their votes in three ways. Global delegation is where members give their vote to a representative on every issue. The second is subject delegation, where people give their vote on specific subjects only, like health or education. The last one is issue delegation, where a member only entrusts another member with their vote on specific issues.

Limitations
The software does have its limitations. In its mission statement, Liquid Feedback says it is an “online system for discussing and voting on proposals in an inner party (or inner organizational) context and covers the process from the introduction of the first draft of a proposal to the final decision.” This means that the software is only intended to be used to decide on policy papers within a party, and is not meant to replace a legislative body’s core function. Germany’s Pirate Party is one of Liquid Feedback’s largest adopters. For now, the software is only used by the party to finalize position papers that then inform decisions at the party’s conventions. Some members would like to see it used to make decisions within the party, but for now, it seems the software is still in its trial period for the Pirate Party. This doesn’t mean only few users have tried out the platform. Almost 10,000 pirates are LQFB members. Yet for now, use of the platform is limited to condensing results and bringing them to a vote at the party’s convention.

There also seems to be a small tech literacy barrier. As is typical of open-source software, the interface and user experience are far from award-winning. Political science professor, Christophe Bieber, says the interface may be “seen as ‘nerdy or geeky’ by many new recruits, especially when compared with the familiar mechanisms of wikis and collaborative text editors. It has an interface only a developer could love”. If Germany, with a high digital literacy might find it a little challenging to gather participants, countries with low digital literacy might be long ways away from adopting such technology.

German Pirate Party Logo

In a NY Times op-ed, Steve Kettmann wrote that on some level, Liquid Feedback “is a gimmick, an effort to get young people interested and involved in the humdrum of German politics, outside the campaign season and even off line. Whatever it is,” Kettmann writes, “ it works: late last month some 1,300 members trekked to the small northern city of Neumünster to elect a new executive board.” Simon Weiss, a Pirate politician in the Berlin Parliament is more sceptical of the idea that the platform might be attracting voters. Weiss says that while the average person might know the Pirate Party is a grassroots movement with a strong internet presence, many are still unfamiliar about LIquid Feedback.

If Liquid Feedback did draw in a crowd, it didn’t last. Last month, German broadcaster Deutsche Welle, reported that the party’s popularity had sunk from 13% to 3% in polls, and four out of five Germans do not believe the party will gain the 5% vote necessary to gain a seat in German parliament. If Liquid Feedback did attract young audiences a year ago, it seems the hype was not sustained through this year and the Pirate Party has major issues to address.

Scalability
Whether this platform can scale or not is difficult to say, and we might have to wait to see more results. Christophe Bieber says the data on the system’s performance remains scarce, so it might be too early to tell. Simone Weiss says Liquid Feedback has always been “intended as a prototype for a future version of democracy” and they are currently experimenting with it themselves. But Liquid Feedbacks problems might be evident already. By October 2012, Der Spiegel wrote “In North Rhine-Westphalia, meanwhile, the Pirate Party’s parliamentarians have used the software to gather general opinions on just two issues so far. A poll of Pirate Party voters there concerning a proposed law to regulate circumcision showed …  20 votes in a federal state with nearly 18 million inhabitants. It’s a grassroots democracy where no one is showing up to participate.” It seems that we cannot know for certain whether the software can scale or not because we have not seen a large enough participation by LQFB members to know for sure.

Democratizing Lawmaking
While no new technology has made it really possible to democratize lawmaking at a large scale, people have certainly been trying. Liquid Feedback isn’t the only software trying to democratize political processes and lawmaking. Germany’s federal parliament is using Adhocracy for a commission on digital policy and the Social Democratic Party (SDP) has been another similar platform for its party think tank.

Liquid Feedback and M5S
In the latter half of last year, the Italian Five Star Movement (M5S) decided to adopt the platform in response to criticism on its lack of internal democracy. Because M5S has a much larger representation in Italy’s senate and chamber of deputies than the Pirate Party does in it’s own country, we might see much wider adoption of the platform in Italy than in Germany. Here is where we might see whether the platform can scale. The issue of security might prove to be the bottleneck in the scaling process. In order to ensure that a person can’t create multiple accounts, vote more than once, or cheat the system, people have to go through a verification process before they are allowed to join. The M5S movement has tens of thousands of members, and verifying all of them and allowing them all to participate in discussions might be a huge undertaking.

MoVimento 5 Stelle Logo

However, the platform has been already adopted at a regional level. Recently, the M5S chapters from the regions of Lombardy and Sicily were able to elect candidates for the presidency using only Liquid Feedback. So far few issues are being discussed in the Lombardy and Sicilian Liquid Feedback portals, presidential candidates being the major one (although the Lazio LQFB seems a bit more active). Yet the platform is still in its infancy and given the success in its ability to chose candidates online, we might expect more issues to be brought up via LQFB.

Online platforms that attempt to democratize political processes such as policy making are still in their infancy. Liquid Feedback is only one such experiment and we can expect to be hearing more of it in the news in the coming years. While it’s shortcomings won’t necessarily mean its demise, if Liquid Feedback doesn’t evolve to solve the challenges of security, scalability, and user experience, then it might end up fading away with all its promises of liquid democracy.

You can try out a test version of Liquid Feedback here and browse the Pirate Party’s version here. For a full list of M5S instances of LQFB, click here.

The Internet vs. The Cartels: Social Media Use in Mexico’s Drug War

May 4th, 2013

Note: This post was made for Clay Shirky’s Political Uses of Social Media class at ITP. It references a couple of documents that the class read for class, but I tried to write this as a self-contained post for those who have not read some of these articles (although they are all linked).

This a brief overview of the use of social media in Mexico and its role in the Mexican Drug War. I will start with a bit of background on how the situation got to where it is, and how the use of social media has evolved throughout the years. There have been several major events that are worth noting which serve to paint a general picture of the role of social media in Mexico during this time of crisis. I will also discuss how the social movements against the cartels and the cartels themselves fit into Jennifer Earl’s Tanks, Tear Gas, and Taxes. Finally, I will look at some possibilities for more effective action by the social movements in Mexico.

Drug cartels have been present in Mexico for several decades, but it wasn’t until President Felipe Calderón, who vowed to wage a war against the cartels, took office in 2006, that the violence became commonplace around the country. There is some debate as to exactly why drug violence rose dramatically during the Calderón years, but most people believe that Calderon’s strategy of eliminating high-ranking cartel members created a power struggle– for control over the organization or over territory–among the different cartels that caused the cartels to become considerably more aggressive. Of course, much of the violence and it’s spilling onto the streets also resulted from the mere fact of sending in thousands of armed forces into cities and towns to wage a war against powerful criminal organizations.

For the first couple of years, traditional news sources such as newspapers would follow the quotidian routine of reporting the horrendous violence on the streets. However, journalists remained unprotected, and the cartels started threatening, and even killing, reporters who wrote about the violence or who tried to expose information the cartels wished to keep secret (a full report on killed or missing journalists is available from the Center to Protect Journalists). Newspapers started to self-censor. In September, 2010, El Diario de Juárez, the largest newspaper from the most violent city, published an op-ed declaring that it had decided to self-censor after the killing of two of it’s journalists. “We ask you to explain what you want from us, what we should try to publish or not publish, so we know what to expect.” the op-ed read. And the trend continued; journalists were threatened or killed, newspapers self-censored, and sometimes even newspaper offices were attacked.

Density Plot of Reporters Killed/Disappeared During the Mexican Drug War

It seems there has been a decline since 2010. Could it be there is less threat to reporters because less reporters are reporting about the violence?

The bloggers started early on (Twitter had not caught on yet). It’s difficult to trace these blogs since some don’t exist anymore, but others are still around. The “first” Blog Del Narco was hosted on Blogspot and started in May, 2008 and remained very active (publishing 1,500+ posts) for about 2.5 years. Just as this blog’s activity began dwindling* (see footnote), the now famous Blog del Narco (BDN) appeared (same name, not related).  Although it’s hard to confirm their motivations since most bloggers remain anonymous, it is believed that these blogs were a response to the media’s self-censorship. Last month, the blogger for BDN agreed to an interview by The Guardian and the Texas Observer, in which she revealed she was a woman in her mid-20’s living somewhere north Mexico. Lucy (as she has nicknamed herself), with the help from a colleague who handles the tech, has posted more than 7,800 posts about the cartels during her 3-year blogging career. She has received multiple death threats from the cartels and has also been accused of spreading the cartel’s message and is therefore also wanted by the police. BDN has since acquired a huge following (3M hits/month) and has become indispensable reading when it comes to narco-news. Lucy sees herself and her work as necessary to fill the void left by the censored media. “If it wasn’t for the blog often bodies wouldn’t be identified”, she says.

But it hasn’t been easy for the bloggers. In 2011, the Zeta cartel started cracking down on those who reported the group’s activities. On September of that year, two people were disemboweled and hung from a bridge by the Zetas with a message warning anyone who decided to blog about their activities. Days later, another blogger was decapitated and left with a similar message. A month later, a fourth blogger. Years later in her interview, Lucy would reveal that some of those bloggers were regular contributors. Since those four killings, there hasn’t been much news about dead bloggers, but the threats nonetheless continue.

Then came the interesting case of Anonymous. I want to add a caveat that there is no way to know if all of this information is 100% true, but this is the widely accepted story of how things went down. During the summer of 2011, the Mexican faction of the loose hacktivist collective Anonymous launched Operation PaperStorm. Anonymous believed that the Veracruz state government was protecting the Zetas while prosecuting those who tweeted about kidnappings (more on this later). After (or perhaps in response to) PaperStorm, a member of Anonymous was kidnapped by the Zetas. Anonymous responded by threatening to release vital information concerning the Zetas cartel if the victim was not released by November 5th. In response, unconfirmed reports suggest that the Zetas recruited computer specialists to try to identify the Anonymous members behind the threat. This digital standoff could’ve been devastating to both groups. The release of such information could have potentially been dangerous to the Zetas cartel, especially in the hands of rival cartels, while the lives of the Anonymous members were at stake if their identities were revealed. As a result of the deadly threat, many Anonymous members denied involvement and disassociated themselves with those who wanted to go forward with the threat. On November 4th, the kidnapped victim was released and the Zetas threatened to kill 10 people for every name released by Anonymous. Both parties backed down.

The Zetas cartel operates as a clandestine operation. Their attacks on bloggers and news media reveal a strategy to control information in order to remain unidentified to ensure the group can continue to operate with impunity. This control of information runs counter to the beliefs of Anonymous, which believes information should be free and decentralized. It was only a matter of time before these two groups collided. The threat of having information about the cartel released posed a great threat to the Zetas’ operations (the information supposedly contained names of members, bribed officials, and messengers). Since Anon operates as a loose collective, a lot of its members were able to disassociate from the attack, leaving Anonymous more susceptible to the Zetas. The fact that these two non-state actors can operate illicitly on the internet is also interesting. Since both groups operate outside of the law, the Mexican government had no choice but to sit back and watch the whole duel unfold. This presents a relatively new dynamic where private actors coerce against each other while remaining relatively undisturbed by a state actor. This confrontation reveals, at least in Mexico, the inability of the government to control their cyberspace and the inability of the government to protect its netizens. Much of these ideas I have presented about the Anon-Zetas standoff come from an article written by Paul Rexton Kan for the Yale Journal of International Affairs. It is extremely interesting, and I suggest you read it.

In his article, Kan presents several interesting and unanswered questions that result from this incident. Kan asks if either groups are aware of a scenario of mutually assured destruction (MAD), where they could both inflict an unacceptable amount of damage to each other. If they are both aware of this MAD scenario, would this deter them from attacking each other again? Since Anonymous is a loose collective, could there be a case where a small faction of the group dissents and decides to attack the Zetas again? With the Zetas gaining technical know-how, will they be able to uncover the identities of Anon? One sure take-away from this conflict is that this type of cyber war undermines the legitimacy of the Mexican government’s authority in cyberspace (and in the offline world as well).

As blogging became increasingly dangerous and more time-intensive, the rest of the netizens took to Twitter. Instead of focusing on denouncing cartel activities, people began using Twitter as a platform to warn people of dangerous situations. The shootouts among cartels and Mexican forces that were so commonplace in Mexico became known as “Risk Situations” (SDR, Situación de Riesgo). Over time people adopted a system of reporting that included city-specific hashtags. If you wanted to monitor SDR’s in Monterrey, you would follow #MTYfollow. Or #ReynosaFollow for Reynosa, etc. Eventually, “social media curators” became beacons for warnings of SDR’s. Andres Monroy-Hernandez, et al. at Microsoft Research has written a great paper on the emergence of these curators, which spend long hours of the day monitoring hashtags and receiving information about SDR’s so that they can relay important information to their followers. Just like the bloggers, these curators say they have stepped up to the plate to be part of the “citizen network to protect and provide tips about civic safety, to avoid becoming victims of crime.”

Facebook pages, such as Valor Por Tamaulipas (VXT), with similar functions as the Twitter curators have sprung up as well. However, just like some blogs, VXT has been threatened. A few months ago, an unknown criminal group posted a $49,000 reward to any information leading to the identity of the VXT admin. Despite the threats, the admin has decided to continue posting. While bloggers and Facebook page admins receive threats, Twitter seems to have been relatively unfazed by the cartels threats. As long as the cartels (or any criminal group) can identify a single person responsible for reporting, they can threaten that person’s life. Since there are hundreds of thousands of Twitter users, the cartels have no way of effectively threatening the Twitter community. This might indicate that in order to launch a relatively safe social media strategy against the cartels, you would need the power of numbers. As one blogger said, “They can’t kill us all.” So far, Twitter information is mostly used to warn about risk situations and not so much to report specific activities from cartels, which might also explain the general disinterest of the cartels towards the use of Twitter. Unfortunately, Twitter is not always as great as it is hyped up to be. In August, 2011 a Twitter user from the state of Veracruz wrote “#verfollow I confirm that in the school ‘Jorge Arroyo’ in the Carranza neighborhood 5 kids were kidnapped, armed group, panic in the zone”. Almost immediately, there was panic all over the city. As a result, the state government tried to prosecute those who tweeted false information (one of the reasons Anon launched Operation PaperStorm). The whole debacle ended making the state government look ridiculous for attempting to label Twitter users as terrorists. This incident does illustrates how difficult it is to trust information from Twitter. The larger and the more anonymous that your network of reporters gets, the less reliable your information becomes. This has been one of the main problems of reporting drug war violence through Twitter almost since its inception. Because people’s reputation is not at stake like it is for a news organization like the New York Times, people have weak motivations to ensure they are providing accurate information. This seems to be the trade-off: a safe and large network with a high incidence of unverifiable and uncurated information, or a small, vulnerable network (often a network of one or two curators) with more reliable information.

In Tanks, Tear Gas, and Taxes, Jennifer Earl refines the theory of repression to include a larger set of variables that have been left out in previous repression studies. She uses three dimensions of repression: the identity of the repressive agent, the repressive action, and whether the action is observable. In our case, we could treat the cartels as the repressive agent and the social media movements or organizations–let’s call them concerned netizens– as the repressed. Under Earl’s ‘Dimensions of Repression‘ (Earl’s terms italicized), the cartels are private agents, who through observable action use coercion. I argue that the cartels’ actions are observable because while we may not know the perpetrator (an unidentifiable agent), the actions are highly visible as exemplified by the accompaniment of written messages with dead bodies, usually as a warning to enemies, and in this case, to the concerned netizens. Earl then argues that some of the combinations to these three dimensions need to be addressed by future research. Research on the threat of observed coercion by private agents, according to Earl, is insufficient.

Yet it’s not that research on the dynamic between the cartels and the concerned netizens is insufficient, our case simply does not fit into any of her approaches to repression. In short, her approaches to repression describe how a private or state actor would react towards a movement depending on certain characteristics of the movement. For example, movements that are weak and threatening will be major targets of repression. The problem is that the cartels simply do not share the same concerns as a state or regular private actor would, mainly because the cartels operate outside of the law and they do not have to be accountable to any constituents. Any threat to the cartels will be met with coercion. The cartels do not need to worry about political opportunities, timing issues, or law-enforcement characteristics, they are free to act as they wish and remain mostly undisturbed by consequences.

I wanted to conclude by providing few ideas (borrowed from Beautiful Trouble) to reflect on. However, when going through the book, I found that most of the tactics, principles, and theories could not be applied to the cartels. The main problem can be exemplified by a short phrase I found in the book: “accountability is what gives democracy it’s bite”. Unfortunately, we’re not dealing with a force that responds to a democratic system, much less a force that has any accountability whatsoever to anyone except its own members, so how can you encourage collective action against such a violent force when the cost of participating is so high? I realize that the class is called “Political Uses of Social Media” and not “Uses of Social Media Against the Cartels”, but it is worth looking at how the theories we have seen in class might apply to this situation. Yet I think the best thing to do is to direct action toward the government. I am not suggesting that people stop blogging or tweeting about crime. Making information public is a way of showing the government’s inability to overpower criminal organizations and a way of holding the government accountable for their deficiencies.  The risk of engaging against the cartels–as Anonymous did–is just too high. The Mexican government is the institution which can be held accountable, and they are the only force large enough to stop this other tremendous force (I use ‘force’ loosely, not advocating for more war). By acting against the government, all the models and theories we have learned in class make sense. We can apply a wide range of tactics to demand change from the government. After all protesting and demanding change is what Mexicans do best.

 

 

*blogspot.blogdelnarco.com stopped posting on Sept. 2010. After a long hiatus, this blogger finally posted again on April 4, 2013: “I wasn’t dead, I was partying. Say what you want, this is the original Blog del Narco”. The timing with the release of ‘Lucy’s’ interview suggests this post was a reaction to ‘Lucy’s” new fame.

Mexican Gov’t Tries to Buy $10M App, Coders Respond by Building It For Free

April 4th, 2013

A few weeks ago, Grupo Reforma reported* the Mexican Chamber of Deputies had signed a contract with an external consulting firm, Pulso Legislativo, to develop a mobile application that would allow Representatives to monitor and publish up-to-date legislative information from their mobile devices for the outrageous sum of about $10 million dollars ($115 million pesos). Making matters worse, was the fact that the Chamber of Deputies wanted to develop this app despite the fact that they already have four main agencies that generate the app’s information, five research centers, and three offices in charge of documentation.

How did the members of the Mexican tech community respond? They created a week-long hackathon and got coders to build an open-source version of the app, for free. The group Codeando Mexico responded to this ludicrous news by setting up the #app115 challenge to which over 160 coders signed up to participate. Tomorrow, Codeando Mexico will present five app submissions at the Legislative Palace of San Lázaro, the same building where the Deputies hold their sessions. For information on tomorrow’s event click here.

Other than ridiculing the Chamber of Deputies who thinks it can get away with trying to buy a $10 million dollar app (not sure if that was actually their intention), Codeando México is trying to highlight the importance of civic participation. It is unfortunate that the government has not yet realized the importance of engaging its citizenry, an effort which might help bridge the gap between the citizens and their representatives (and potentially save a lot of money). Hopefully initiatives like Codeando Mexico will gather more attention in the near future. I would love to see more coders getting together on a Saturday night and coding “over some tequilas”.

 *Linked to different article, Grupo Reform has a paywall.

Update: To read more about this, Eric Tecayehuatl, has covered this on Gizmodo (In Spanish).

De-anonymizing Stop and Frisk Data.

January 2nd, 2013

I started with the premise that 87% of Americans are uniquely identifiable by knowing their date of birth, zip code, and gender. The Stop and Frisk (SNF) data gives you date of birth, precinct, gender, race, height, weight, eye color, hair color, and build. The original SNF data set contains 685,724 stops for 2011. However, out of those stops, only 2/3 had valid dates of birth. By ‘valid’ I mean, between the ages of 0 to 112 (around 275,000 stops where of people born on Dec. 31, 1900). Since date of birth is crucial to de-anonymization, I excluded those data points from the analysis. My numbers will therefore differ from the NYCLU’s report, since they did include these entries.

After cleaning the data a bit, I chose to only use D.O.B, gender, race, precinct, and height to de-anonymize the data. I did not choose the rest of the descriptors because the police officer conducting the stop might not always enter the same information for the same person. First, only in 55% of stops did the suspect provide a photo ID which could provide accurate details of their weight, hair color, etc. The police officer would have had to guess all of the person’s information correctly every time for the other 45% of stops. Second, people’s weight, build and hair color can change over the year or not easily identifiable at night. Lastly, I realize that height also, changes, especially in people below 20 years, but I wanted to play it a little safe somehow. I thought height would be easy for a police officer to guess correctly, so I kept height. Using these descriptors, I found 364,706 unique individuals. 22,649 of whom were stopped more than once. Here are the top 20 people stopped in 2011, and the number of times they have been stopped.

Screen Shot 2012-12-20 at 7.44.34 PM

The string of numbers is the person’s “name”. From left to right, the numbers mean precinct, gender, race, DOB, and height. You’ll notice that 18/20 precincts are precincts 60, 61, and 101 (Coney Island, Gravesend, and Far Rockaway) I’ll write more on these later, but first some numbers on people stopped more than once.

  • 6.2% of people stopped where stopped more than once (22,476 out of 364,706).
  • 60.7% of people stopped more than once where black.
  • 29.0% were hispanic.
  • 7.9% were white.
  • 2.4% were others (Asians, Pacific Islanders, Native Americans, Others)

Going back to precincts 60, 61, and 101. After I first noticed that the overwhelming presence of these three precincts in the top 20 list, I mapped out all the people who had been stopped more than once and got a map with points pretty much all over New York City. Notice, each dot represents a person, not a stop. The position of the person is the average position of all the person’s stops.

Then I mapped out everyone stopped more than 5 times.
People Stopped More Than 5 Times

Out of the 22,000+ people stopped more than once, there where 340 that were stopped more than 5 times. Here is a table of the top 10 precincts with people stopped more than 5 times.

Area/Neighborhood Precinct # People Stopped > 5 Times
Far Rockaways 101 83
Sheepshead Bay 61 58
East New York 75 24
Williamsburg 90 23
Coney Island 60 21

These top 5 precincts contain 61% of the people stopped more than 5 times. It would be interesting to find out what is going on there, but there doesn’t seem to be an evident explanation. So far I have not found a common characteristic that these precincts share. Here are some facts about the 340 people that have been stopped more than 5 times:

  • Precinct 101 accounts for 1/4 of the 340 people stopped more than 5 times.
  • One woman was stopped 14 times in Precinct 61 (Sheepshead Bay)
  • 72% of people stopped > 5 times were African American/Black
  • 13% were Hispanic
  • 15% were White
  • The average age is 24.7 years (max 56, min 16)
  • These 340 people make up 2,686 stops.
  • 71% of those stops included a frisk.
  • Less than 5% (124) of stops led to an arrest.
  • Only 7 of those arrests were because of the criminal possession of a weapon (0.26%).

I don’t expect these numbers to be an accurate representation of all multiple stops in NYC. However, I do think that 1) they reveal a pattern, and 2) these numbers are a best case scenario, and in fact, I think the real numbers are way worse. After all, we know that there is at least one person who was stopped more than 60 times before he turn 18.

If you would like to read about the Top 10 most stopped individuals in New York City, check out the comic book I made for my Data Rep class.