Using_the_Mindat_API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qrMhTOpOyjAP"
   },
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/ChuBL/How-to-Use-Mindat-API/blob/main/How_to_Use_Mindat_API.ipynb\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
    "</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u65PipxXuNbO"
   },
   "source": [
    "# How to use the Mindat API to access data from Mindat\n",
    "\n",
    "Stephen M. Richard\n",
    "2023-06-20\n",
    "\n",
    "Based on tutorial by Jiyin Zhang\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OOT8B2v7odok"
   },
   "source": [
    "## 0. Access Your Mindat API Token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "59gsuIxcoiIU"
   },
   "source": [
    "[How to Get My Mindat API Key or Token?](https://www.mindat.org/a/how_to_get_my_mindat_api_key)\n",
    "\n",
    "Check the documentation at [Mindat API doc](https://api.mindat.org/schema/redoc/) to see if there have been updates to the API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YmvOE3ClX207"
   },
   "source": [
    "## 1. Dependencies (Run this section first)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UublqFDuYxeU"
   },
   "source": [
    "This set of examples runs with files that are located in the file directory with the python notebook .ipynb file, for running with Jupyter or similar host in your local desktop environment with an active internet connection. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gpxJUYWHaPCw"
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import os\n",
    "import sys\n",
    "import json\n",
    "import requests\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RhAUoA9pcIQv"
   },
   "source": [
    "You should **avoid** placing your API token explicitly in your code if you plan to share it. That would include working with a notebook that is in a public Github repo, like this one.\n",
    "\n",
    "The solution here is to have the api key saved in a file accessible from your notebook environment (e.g. in the same directory), and adding that file to your github .ignore file so it is not copied to the public Github repo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "nW9tqw7ScEs9"
   },
   "outputs": [],
   "source": [
    "api_key_file_dir = \"local/api_key.txt\"\n",
    "try:\n",
    "    with open(api_key_file_dir, 'r') as f:\n",
    "        YOUR_API_KEY = f.read()\n",
    "except FileNotFoundError:\n",
    "    print(\"API key file not found. Please create a text file containing your api key and place it in the correct directory.\")\n",
    "    \n",
    "try:\n",
    "    assert 0 != len(YOUR_API_KEY)\n",
    "except AssertionError:\n",
    "    raise Exception(\"Please set a valid API token before the start!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UEMu5YSJwdC2"
   },
   "source": [
    "### Basic access pattern"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RrIOSXhLwdC2"
   },
   "outputs": [],
   "source": [
    "# API root entry point\n",
    "MINDAT_API_URL = \"https://api.mindat.org\"\n",
    "\n",
    "# authorization header that must be included with each request.\n",
    "headers = {'Authorization': 'Token '+ YOUR_API_KEY}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# see https://api.mindat.org/schema/redoc/ for documentation on using the Mindat API\n",
    "\n",
    "\n",
    "# using the geomaterials_search endpoint. this executes a text search using the term in the \n",
    "#  q parameter, and returns a fixed set of fields. \n",
    "# Apparently these are id','name': ,'synid','ima_status', 'ima_approved'\n",
    "# other parameters besides 'q' are ignored\n",
    "\n",
    "filter_dict = {\n",
    "    'q':'dunite',\n",
    "}\n",
    "\n",
    "params = filter_dict\n",
    "\n",
    "# use python requests package to GET results from mindat\n",
    "response = requests.get(MINDAT_API_URL+\"/geomaterials_search/\",\n",
    "                    params=params,\n",
    "                    headers=headers)\n",
    "# assume the query succeeds. See later examples for using the response.status_code\n",
    "#  to check if the request worked.\n",
    "\n",
    "#  another handy requests package function to cast the results to JSON.\n",
    "json_file = response.json()\n",
    "\n",
    "# this will echo the json file containing the query results.\n",
    "json_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Get information using a list of Mindat IDs\n",
    "This example uses the geomaterials endpoint (used to be called items).  The fields parameter specifies the fields that will be returned in the response.\n",
    "\n",
    "There is a long list of parameters that can be used with this endpoint in the doco\n",
    "at [geomaterials](https://api.mindat.org/schema/redoc/#tag/geomaterials). See the 'fields' parameter for alist of the fields that can be returned.\n",
    "\n",
    "Note that wildcards will work with the name parameter. '*' for one or more character, '_' for any single character.\n",
    "\n",
    "#### examples \n",
    "https://api.mindat.org/geomaterials/?name=baryo*\n",
    "(name begins with baryo)\n",
    "\n",
    "https://api.mindat.org/geomaterials/?name=__r______&elements_inc=U&fields=id,name,mindat_formula\n",
    "\n",
    "(Uranium minerals, name contains 9 letters and 3rd letter is r)\n",
    "\n",
    "https://api.mindat.org/geomaterials/?name=quartz\n",
    "(Exact match, case and diacretic insensitive)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "KUus01VVnrLn"
   },
   "outputs": [],
   "source": [
    "#repeating for convienence\n",
    "\n",
    "MINDAT_API_URL = \"https://api.mindat.org\"\n",
    "headers = {'Authorization': 'Token '+ YOUR_API_KEY}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xpu636_dpIWs"
   },
   "outputs": [],
   "source": [
    "# set your selected fields here\n",
    "# fields_str = 'id','longid','guid','name','entrytype','entrytype_text','description_short','rock_parent','rock_parent2','rock_root','rock_bgs_code','meteoritical_code'\n",
    "\n",
    "fields_str = 'id','longid','name', 'updttime'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set up a list of the IDs you want to get information about. \n",
    "# e.g. \n",
    "idlist = ['11263','48145','49089','49091','49093','49504','49505','49506','49507','49508','49509','49510','49511','49512','49513','49514','49517','49518','49519','49520','49521','49522','49523','49524','49525','49526','49527','49528','49529','49530','49531','49532','49533','49534','49535','49536','49537','49538','49539','49540','49541','49542','49543','49544','49545','49546','49547','49548','49549','49550','49551','49552','49553','49554','49556','49557','49558','49559','49560','49561','49562','49563','49564','49565','49566','49567','49568','49569','49570','49571','49572','49573','49574','49575','49576','49577','49578','49579','49580','49581','49584','49585','49586','49587','49588','49589','49590','49591','49592','49593','49595','49596','49597','49599','49600','49603','49604','49606','49607','49608','49609','49610','49612','49614','49615','49616','49617','49618','49619','49620','49621','49622','49623','49625','49626','49627','49628','49629','49630','49631','49632','49633','49634','49635','49636','49637','49638','49639','49640','49641','49642','49643','49644','49645','49646','49647','49648','49649','49650','49651','49652','49653','49654','49655','49656','49657','49658','49659','49660','49661','49662','49663','49664','49665','49666','49667','49668','49669','49670','49671','49672','49673','49674','49675','49676','49677','49678','49679','49680','49682','49683','49684','49685','49686','49688','49689','49690','49691','49692','49693','49695','49696','49697','49698','49699','49700','49702','49703','49705','49707','49708','49709','49710','49711','49712','49713','49715','49716','49717','49718','49719','49720','49721','49722','49723','49724','49725','49726','49727','49728','49729','49730','49731','49732','49733','49734','49735','49736','49737','49738','49739','49740','49741','49742','49743','49744','49745','49746','49750','49751','49752','49753','49754','49755','49756','49757','49758','49760','49761','49762','49763','49764','49765','49767','49768','49769','49770','49772','49773','49774','49776','49777','49778','49779','49780','49781','49782','49783','49784','49785','49786','49787','49788','49789','49790','49791','49792','49793','49794','49795','49796','49797','49798','49799','49800','49802','49803','49804','49805','49806','49807','49808','49809','49810','49811','49812','49813','49814','49815','49816','49817','49818','49819','49820','49821','49822','49823','49824','49825','49826','49827','49828','49829','49831','49832','49833','49835','49836','49837','49838','49839','49840','49841','49842','49843','49844','49845','49846','49847','49848','49849','49850','49851','49852','49853','49854','49856','49857','49859','49860','49861','49862','49863','49864','49865','49866','49867','49869','49872','49873','49876','49877','49878','49879','49880','49881','49882','49883','49884','49885','49886','49887','49888','49889','49890','49891','49892','49893','49894','49895','49896','49898','49899','49901','49902','49904','49905','49906','49907','49908','49909','49910','49911','49912','49913','49914','49915','49916','49917','49918','49919','49920','49921','49922','49923','49924','49925','49926','49927','49928','49929','49930','49931','49932','49933','49934','49935','49936','49937','49938','49939','49941','49942','49944','49945','49947','49950','49951','49952','49953','49954','49955','49956','49957','49958','49959','49960','49961','50269','50270','50444','50445','50446','50447','50448','50449','50450','50451','50452','50453','50454','50455','50456','50457','50458','50459','50460','50461','50462','50463','51453','52197','52198','52199','52200','52201','52202','52203','52204','52205','52206','52207','52208','52209','52210','52211','52212','52213','52214','52215','52216','52217','52218','52219','52220','52221','52222','52223','52224','52225','52226','52227','52228','52229','52237','52238','52239','52240','52241','52360','52368','52369','52370','52371','52372','52373','52374','52393','52394','52395','52396','52720','52787','54116','54117']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Query for items by id in an id list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "j7z0PF1WpVC1",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas\n",
    "import time\n",
    "\n",
    "json_array = []\n",
    "\n",
    "idlist = ['11263','48145'] # short list for testing\n",
    "\n",
    "for idstr in idlist:\n",
    "\n",
    "    params = {\n",
    "        'fields': fields_str, # put your selected fields here\n",
    "        'id__in': idstr, # set the item amount for each page\n",
    "        'format': 'json'\n",
    "    }\n",
    "\n",
    "    response = requests.get(MINDAT_API_URL+\"/geomaterials/\",\n",
    "                    params=params,\n",
    "                    headers=headers)\n",
    "\n",
    "# do some simple error testing to see if the response was successful\n",
    "    if 200 <= response.status_code <= 299:\n",
    "        json_out = response.json()\n",
    "    #    print (json_out)\n",
    "        json_array = json_array + json_out[\"results\"]\n",
    "    else:\n",
    "        print ('problem-- ', idstr)\n",
    "\n",
    "# NOTE that the 'results' key in the response is an array of json objects containing the fields \n",
    "#   specified in the paramters. The other keys in the response are 'previous' and 'next';\n",
    "#   these are used for paging if the list of results is very long. See example of using these\n",
    "#   below. \n",
    "        \n",
    "# if your list is long, add a little delay so you don't bog down the server        \n",
    "    time.sleep(3)\n",
    "    \n",
    "    \n",
    "print ('Done') \n",
    "\n",
    "#this will echo what you got\n",
    "json_array\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using the Cursor for pagination"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# get all records that have a mereoritical_code value, using the geomaterials endpoint\n",
    "# have to use cursor pagination\n",
    "\n",
    "import pandas\n",
    "import time\n",
    "\n",
    "MINDAT_API_URL = \"https://api.mindat.org\"\n",
    "headers = {'Authorization': 'Token '+ YOUR_API_KEY}\n",
    "\n",
    "#fields_str = 'id','longid','guid','name'\n",
    "fields_str = 'id','longid','guid','name','entrytype','entrytype_text','description_short','rock_parent','rock_parent2','rock_root','rock_bgs_code','meteoritical_code'\n",
    "\n",
    "json_array = []\n",
    "\n",
    "# using field added by Pavel Martynov to check if there is a meteoritical_code value\n",
    "# https://mindat.slack.com/archives/C02EMJN8658/p1686930757175029\n",
    "params = {\n",
    "    'fields': fields_str, # put your selected fields here\n",
    "    'format': 'json',\n",
    "    'meteoritical_code_exists':'true'\n",
    "}\n",
    "\n",
    "#seed response to get the ball rolling\n",
    "response = requests.get(MINDAT_API_URL+\"/geomaterials/\",\n",
    "                params=params,\n",
    "                headers=headers)\n",
    "\n",
    "if 200 <= response.status_code <= 299:\n",
    "    json_out = response.json()\n",
    "#    print (json_out)\n",
    "    json_array = json_array + json_out[\"results\"]\n",
    "else:\n",
    "    print ('problem')\n",
    "\n",
    "json_out[\"next\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {}\n",
    "\n",
    "# The 'next' key in the response is a URL with the field list included, and a 'cursor'\n",
    "#  parameter that provides the authentication token and identifies the next response\n",
    "#  segment to send. When this parameter is 'None', all the results have been sent. \n",
    "#  There doesn't seem to be a way to alter then number of result objects in each\n",
    "#  increment-- its alwas 10, so be prepared to wait awhile if you're pulling much data.\n",
    "# this example pulls about 450 records, so takes 45 or so iterations. \n",
    "\n",
    "while json_out[\"next\"] is not None :\n",
    "    response = requests.get(json_out[\"next\"],\n",
    "                params=params,\n",
    "                headers=headers)\n",
    "    \n",
    "    print (response.status_code)\n",
    "    if 200 <= response.status_code <= 299:\n",
    "        json_out = response.json()\n",
    "    #    print (json_out)\n",
    "        json_array = json_array + json_out[\"results\"]\n",
    "    else:\n",
    "        print ('problem-- ', json_out[\"next\"])\n",
    "\n",
    "print ('Done')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the results are accumulated in a JSON array; convert to a pandas frame\n",
    "# you could do analysis using that frame. In this example, it just gets \n",
    "# dumped to a local csv file to open in Excel. \n",
    "\n",
    "df_result = pandas.DataFrame(json_array)\n",
    "\n",
    "\n",
    "# don't forget to put an appropriate output file name here\n",
    "df_result.to_csv('outfile.csv') \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an aside, if all your results come back in one response, you can \n",
    "Create a DataFrame direcly from the results array in the response like this:\n",
    "\n",
    "df_nested_list = pandas.json_normalize(json_array, record_path =['results'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# About those long ID strings ...\n",
    "\n",
    "If you look at the mindat pages, you'll see that each rock, mineral, locality etc has an integer id. \n",
    "\n",
    "This ID shows up in the URLs for the Mindat pages, e.g. \n",
    "https://www.mindat.org/loc-19174.html is about the Beltana Mine in South Australia. Its ID is 19174. \n",
    "These integers are autoincremented for different resources, in particular localities and rock/mineral. Thus there might be a locality and a mineral with the same ID number. To create a unique identifier for any mindat resource, a 'long ID' is constructed that looks like this: \n",
    "\n",
    "1:2:19174:9\n",
    "\n",
    "It consists of 4 segments, delimited by colon (':') characters. \n",
    "\n",
    "- The first segment is a digit that identifies the assigning authority; 1 is for Mindat. I'm not sure what other values there are...\n",
    "- The second segement is a digit that identifies the resource type 1- mineral, rocks, meteorites, 2- locality \n",
    "- The third segment is an integer with 1 to 5 digits;  the mindat id (same number that shows up in the URL)\n",
    "- The last segment is a single digit that is a checksum calculated from the preceding three segments, included for error checking. The algorithm is shown in the following cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# mindat check sum algorithm\n",
    "# from Jolyon,2023-06-18\n",
    "\n",
    "def mindat_longid(authority, type, id):\n",
    "    out = \"{}:{}:{}:\".format(authority, type, id)\n",
    "    out2 = \"{}{}{}\".format(authority, type, id)\n",
    "    t = 0\n",
    "    for i in range(len(out2)):\n",
    "        if i % 2 == 1:\n",
    "            t += int(out2[i]) * 3\n",
    "        else:\n",
    "            t += int(out2[i])\n",
    "    ck = t % 10\n",
    "    if ck:\n",
    "        ck = 10 - ck\n",
    "    out += str(ck)\n",
    "    return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run checksum function\n",
    "mindat_longid(1,2,19174)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}