130507-Berkeley-iSchool-OpenData.ipynb

{
 "metadata": {
  "name": "130507-Berkeley-iSchool-OpenData"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Exploring Open Data with Pandas and IPython at the Berkeley I School"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "\"Working with Open Data\", a course by Raymond Yee"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "This will be a guest post, authored by [Raymond Yee](http://www.ischool.berkeley.edu/people/faculty/raymondyee) from the [UC Berkeley School of Information](http://www.ischool.berkeley.edu) (or I School, as it is known around here).  This spring, Raymond has been teaching a course titled [\"Working with Open Data\"](http://www.ischool.berkeley.edu/courses/i290t-wod), where students learn how to work with openly available data sets with Python.\n",
      "\n",
      "Raymond has been using IPython and the notebook since the start of the course, as well as hosting lots of materials directly using github.  He kindly invited me to lecture in his course a few weeks ago, and I gave his students an overview of the IPython project as well as our vision of reproducible research and of building narratives that are anchored in code and data that are always available for inspection, discussion and further modification.\n",
      "\n",
      "Towards the end of the course, his students had to develop a final project, organizing themselves in groups of 2-4 and producing a final working system that would use open datasets to produce an interesting analysis of their choice.\n",
      "\n",
      "I recently had the chance to see the final projects, and I have to say that I walked out blown away by the results.  This is a group of students who don't come from traditionally computationally-intensive backgrounds, as the only requirement was some basic Python experience.  And in a matter of a few weeks, they created very compelling tools, that dove into different problem domains from health care to education and even sports, producing interesting results complete with a narrative, plots and even interactive JavaScript controls and SVG output elements.  Keep in mind that the tools to do some of this stuff aren't even really documented or explained much in IPython yet, as we haven't really dug into that part in earnest (that is our [Fall 2013 plan](https://github.com/ipython/ipython/wiki/Roadmap:-IPython)).\n",
      "\n",
      "The students obviously did run into some issues, and I took notes on what we can do to improve the situtaion. We had a follow-up meeting on campus where we gave them pointers on how to do certain things more easily. But to me, these results validate the idea that we can construct [computational narratives based on code and data](http://blog.fperez.org/2013/04/literate-computing-and-computational.html) that will ultimately lead to a more informed discourse.\n",
      "\n",
      "I am very much looking forward to future collaborations with Raymond: he has shown that we can create an educational experience around data-driven discovery, using IPython, pandas, matplotlib and the rest of the SciPy stack, that is completely accessible to students without major computational training, and that lets them produce interesting results around socially interesting and relevant datasets.\n",
      "\n",
      "I will now pass this on to Raymond, so he can describe a little bit more about the course, challenges they encountered, and highlight some of the course projects that are already available on github for further forking and collaboration.\n",
      "\n",
      "As always, this post is available [as an IPython notebook](http://nbviewer.ipython.org/urls/raw.github.com/fperez/blog/master/130507-Berkeley-iSchool-OpenData.ipynb). *The rest of the post is Raymond's authorship*."
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "The course"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Over a [15-week period](http://htmlpreview.github.io/?https://github.com/rdhyee/working-open-data/blob/master/lectures/day01.html#(8)), my students and I met twice a week to study open data, using Python to access, process, and interpret that data. Twenty-four students completed the course: 17 Masters students from the I School and 7 undergraduates from electrical engineering/computer science, statistics, and business.   \n",
      "\n",
      "We covered about half of our textbook [*Python for Data Analysis*](http://shop.oreilly.com/product/0636920023784.do) by [Wes McKinney](http://blog.wesmckinney.com/).  Accordingly, a fair amount of our energy was directed to studying the [pandas](http://pandas.pydata.org/) library.  The prerequisite programming background was the [one semester Python minimum requirement for I School Masters students](http://www.ischool.berkeley.edu/courses/i90). The students learned a lot (while having a good time overall, so I'm told) about how to program pandas and use the IPython notebook by working through a [series of assignments](https://piazza.com/berkeley/spring2013/info290t/resources). Students filled in missing code in IPython notebooks I had created to illustrate various techniques for data analysis. (Most of the resources for the course are contained in the [course github repository](https://github.com/rdhyee/working-open-data).)\n",
      "\n",
      "My students and I were particularly grateful for the line-up of guest speakers (which included Fernando Perez), who shared their expertise on topics ranging from [scraping the web to archive legal cases](https://docs.google.com/file/d/0B-NkkRVaYqkUUFRfNXFhWnIydXM/edit?pli=1), the [UC Berkeley Library Data Lab](http://www.lib.berkeley.edu/wikis/datalab/), [open data in archaeology](http://opencontext.org/), [open access web crawling](http://www.slideshare.net/davelester/introduction-to-common-crawl), and [scientific reproducibility](https://docs.google.com/file/d/0B-NkkRVaYqkUcGE2Q0tYSmlKdXM/edit). "
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Final projects"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The culmination of the course came in the final projects, where groups of two to four students designed and implemented data analysis projects. The final deliverable for the course was an IPython notebook that was [expected to contain the following attributes](http://htmlpreview.github.io/?https://github.com/rdhyee/working-open-data/blob/master/lectures/day27.html#(5)):\n",
      "\n",
      "* a clear articulation of the problem being addressed in the project\n",
      "* a clear description of what was originally proposed and what the students ended up doing, describing what led the students to go from the proposed work to the final outcome\n",
      "* a thorough description of what was behind-the-scenes: the sources of data from which the students drew and the code they wrote to analyze the data\n",
      "* a clear description of the results and precise details on how to reproduce the results\n",
      "* if the students were to continue their project beyond the course, what would be this future work\n",
      "* a paragraph outlining how the work was split among group members.\n",
      "\n",
      "\n",
      "The students and I welcomed enthusiastic visitors to the [Open House](http://www.ischool.berkeley.edu/newsandevents/events/20130507opendataexhibition) -- in which the I School community and, in fact, the [larger campus community was invited to attend](http://events.berkeley.edu/?event_ID=66786&date=2013-05-07&tab=all_events). \n",
      "\n",
      "<a href=\"http://www.flickr.com/photos/raymondyee/8727830281/\" title=\"Working with Open Data 2013 Open House by Raymond Yee, on Flickr\"><img src=\"http://farm8.staticflickr.com/7353/8727830281_d96863791e.jpg\" width=\"500\" height=\"282\" alt=\"Working with Open Data 2013 Open House\"></a>\n",
      "\n",
      "<a href=\"http://www.flickr.com/photos/raymondyee/8727829449/\" title=\"Working with Open Data 2013 Open House by Raymond Yee, on Flickr\"><img src=\"http://farm8.staticflickr.com/7428/8727829449_db9c54327d.jpg\" width=\"500\" height=\"282\" alt=\"Working with Open Data 2013 Open House\"></a>"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here are abstracts for the eight projects, where each screenshot is a link to its IPython notebook. Enjoy!\n",
      "\n",
      "**Stock Performance of Product Releases** <br/>\n",
      "Edward Lee, Eugene Kim\n",
      "\n",
      "<a href=\"http://nbviewer.ipython.org/5536678\"><img src=\"https://raw.github.com/fperez/blog/master/fig/ischool-yee-nb-01.png\"></a>\n",
      "\n",
      "Drawing connections for open data available pertaining to Apple in order to examine how Apple's stock performance was impacted by a certain product. We examine Wikipedia data for detailed information on Apple's product releases,\n",
      "make use of Yahoo Finance's API for specific stock performance metrics, and openly available Form 10-Q's for internal (financial) changes to Apple. The main purpose is to examine available data to draw new conclusions centered on the time around the product release date.\n",
      "\n",
      "\n",
      "**Education First** <br/>\n",
      "Carl Shan, Bharathkumar Gunasekaran, Haroon Rasheed Paul Mohammed, Sumedh Sawant\n",
      "\n",
      "<a href=\"http://nbviewer.ipython.org/5530732\"><img src=\"https://raw.github.com/fperez/blog/master/fig/ischool-yee-nb-02.png\"></a>\n",
      "\n",
      "Most parents nowadays have a general sense of the significant factors for choosing a school for their children. However, with a lack of existing tools and information sources, most of these parents have a hard time measuring, weighing and comparing these factors in relation to geographical areas when they are trying to pick the best place to live in with the best schools.\n",
      "\n",
      "Thus, our team aims to address this problem by visualizing the statistical data from the NCES with geo-data to help the parents through the process of picking the best area to live in with the best schools. Parents can specify exactly what parameters they consider important in their decision process and we will generate a heat-map of the state they\u2019re interested in living in and dynamically color it according to how closely each county matches their preferences. The heat map will be displayed with a web browser.\n",
      "\n",
      "\n",
      "**The League of Champions** <br/>\n",
      "Natarajan Chakrapani, Mark Davidoff, Kuldeep Kapade\n",
      "\n",
      "<a href=\"http://nbviewer.ipython.org/5494071\"><img src=\"https://raw.github.com/fperez/blog/master/fig/ischool-yee-nb-03.png\"></a>\n",
      "\n",
      "In the soccer world, there is a lot of money involved in transfer of players in the premier leagues around the world. Focusing on the English Premier League, our project - \u201cThe League Of Champions\u201d aims to analyze the return on investment on soccer transfer done by teams in the English premier league. It aims to measure club return on each dollar spent on their acquired players for a season, on parameters like Goals scored, active time on the field, assists etc. In addition, we also look to analyze how big a factor player age is, in commanding a high transfer fee, and if clubs prefer to pay large amounts for specialist players in specific field positions.\n",
      "\n",
      "\n",
      "**All About TEDx** <br/>\n",
      "Chan Kim, JT Huang\n",
      "\n",
      "<a href=\"http://nbviewer.ipython.org/5439852\"><img src=\"https://raw.github.com/fperez/blog/master/fig/ischool-yee-nb-04.png\"></a>\n",
      "\n",
      "TED is a nonprofit devoted to Ideas Worth Spreading. It started out (in 1984) as a conference bringing together people from three worlds: Technology, Entertainment, Design. The TED Open Translation Project brings TED Talks beyond the English-speaking world by offering subtitles, interactive transcripts and the ability for any talk to be translated by volunteers worldwide. The project was launched with 300 translations, 40 languages and 200 volunteer translators; now, there are more than 32,000 completed translations from the thousands-strong community. The TEDx program is designed to give communities the opportunity to stimulate dialogue through TED-like experiences at the local level.\n",
      "\n",
      "Our project wants to encourage people to translate TEDx Talk as well by showing how TEDx Talk videos are translated and spreaded among different languages, places and topics, and comparing the spreading status with TED Talk videos.\n",
      "\n",
      "\n",
      "**Environmental Health Gap** <br/>\n",
      "Rohan Salantry, Deborah Linton, Alec Hanefeld, Eric Zan\n",
      "\n",
      "<a href=\"http://nbviewer.ipython.org/urls/raw.github.com/ahane/airquality_asthma/master/Health_indicators.ipynb\"><img src=\"https://raw.github.com/fperez/blog/master/fig/ischool-yee-nb-05.png\"></a>\n",
      "\n",
      "There is growing evidence to support environmental factors trigger chronic diseases such as asthma that result in billions in health care costs.  However a gap in knowledge exists concerning the extent of the link and where it is more prevalent.  We aim to create a framework for closing this gap by integrating health and environmental condition data sets.   Specifically, the project will link emissions data from the EPA and the California Department of Public Health in an attempt to find a correlation between incidences of asthma treatments and emissions seen as triggers for asthma.  \n",
      "\n",
      "The project hopes to be a stepping stone for policy decisions concerning the value tradeoff between health care treatment and environmental regulation as well as where to concentrate resources based on severity of need. \n",
      "\n",
      "\n",
      "**World Bank Data Analysis** <br/>\n",
      "Aisha Kigongo, Sydney Friedman, Ignacio P\u00e9rez\n",
      "\n",
      "<a href=\"http://nbviewer.ipython.org/urls/raw.github.com/iaperez/WWOD/master/code/WorldBankData.ipynb\"><img src=\"https://raw.github.com/fperez/blog/master/fig/ischool-yee-nb-06.png\"></a>\n",
      "\n",
      "Our goal was to use a variety of tools to investigate the impact of project funding in developing countries.  In order to do so, we looked at open data from the World Bank, which keeps a strong track of every project that gets funded, who funds it, and the goal of the project whether agricultural, economic or related to health.  By using Python, we used an index of the World Bank to see where the most funded countries were and how they related to various indicators such as the Human Development Index, the Freedom Index, and for the future, health, educational and other economic indexes.  Our secondary goal is to analyze what insight open data can give us as to how effective initiatives and funding actually is as opposed to what it\u2019s meant to be. \n",
      "\n",
      "\n",
      "**Dr. Book** <br/>\n",
      "AJ Renold, Shohei Narron, Alice Wang\n",
      "\n",
      "<a href=\"http://nbviewer.ipython.org/urls/raw.github.com/AJRenold/doctorbook/master/notebooks/DocBookParse.ipynb\"><img src=\"https://raw.github.com/fperez/blog/master/fig/ischool-yee-nb-07.png\"></a>\n",
      "\n",
      "\n",
      "When we read a book, all the information is contained in that resource. But what if you could learn more about a concept, historical figure, or location presented in a chapter? Dr. Book expands your reading experience by connecting people, places, topics and concepts within a book to render a webpage linking these resources to Wikipedia.\n",
      "\n",
      "\n",
      "**Book Hunters** <br/>\n",
      "Fred Chasen, Luis Aguilar, Sonali Sharma\n",
      "\n",
      "<a href=\"http://nbviewer.ipython.org/5552945\"><img src=\"https://raw.github.com/fperez/blog/master/fig/ischool-yee-nb-08.png\"></a>\n",
      "\n",
      "\n",
      "When we search for books on the internet we are often overwhelmed with results coming from various sources. It\u2019s difficult to get direct trusted urls to books. Project Gutenberg, HathiTrust and Open Library all provide an extensive library of books online, each with their own large repository titles. By combining their catalogs, Book Hunters enables querying for a book across those different sources, our project will highlight key statistics about the three datasets. These statistics include: number of books in all the three data sources, formats, language, publishing date.  Apart from that we will ask users to search for a particular book of interest and we will return combined results from all the three resources and also provide the direct link to the pdf, text or epub format of the book. This will be an exercise to filter out results for the users and provide them with easy access to the books that they are looking for."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "\n"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}