api-entities/sources/filter-sources.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "[![GitHub Repository](https://img.shields.io/badge/GitHub-Repository-181717?style=for-the-badge&logo=GitHub&link=https://github.com/Mearman/openalex-docs/tree/main)](https://github.com/Mearman/openalex-docs/tree/main)[![Open in GitHub](https://img.shields.io/badge/Open%20in-GitHub-181717?style=for-the-badge&logo=github&link=https://github.com/Mearman/openalex-docs/blob/main/api-entities/sources/filter-sources.ipynb)](https://github.com/Mearman/openalex-docs/blob/main/api-entities/sources/filter-sources.ipynb)[![Open in Colab](https://img.shields.io/badge/Open%20in-Colab-F9AB00?style=for-the-badge&logo=Google%20Colab&link=https://colab.research.google.com/github/Mearman/openalex-docs/blob/main/api-entities/sources/filter-sources.ipynb)](https://colab.research.google.com/github/Mearman/openalex-docs/blob/main/api-entities/sources/filter-sources.ipynb)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "%pip install --upgrade \"git+https://github.com/Mearman/openalex-python-pydantic-v1.git\"\n",
        "%pip install pandasai"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import json\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from openalex_api import Configuration, ApiClient, AutocompleteApi, AuthorsApi, ConceptsApi, FundersApi, InstitutionsApi, PublishersApi, SourcesApi, WorksApi\n",
        "\n",
        "configuration = Configuration(host=\"https://api.openalex.org\")\n",
        "autocomplete_api = AutocompleteApi(ApiClient(configuration))\n",
        "authors_api = AuthorsApi(ApiClient(configuration))\n",
        "concepts_api = ConceptsApi(ApiClient(configuration))\n",
        "funders_api = FundersApi(ApiClient(configuration))\n",
        "institutions_api = InstitutionsApi(ApiClient(configuration))\n",
        "publishers_api = PublishersApi(ApiClient(configuration))\n",
        "sources_api = SourcesApi(ApiClient(configuration))\n",
        "works_api = WorksApi(ApiClient(configuration))\n",
        "\n",
        "from pandasai import SmartDataframe\n",
        "from pandasai.llm import OpenAI"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title  { run: \"auto\", display-mode: \"form\" }\n",
        "openapi_token = \"\" # @param {type:\"string\"}"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Filter sources\n",
        "\n",
        "You can filter sources with the `filter` parameter:\n",
        "\n",
        "* Get sources that have an ISSN\n",
        "  [`https://api.openalex.org/sources?filter=has_issn:true`](https://api.openalex.org/sources?filter=has_issn:true)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title { run: \"auto\", vertical-output: false }\n",
        "# https://api.openalex.org/sources?filter=has_issn:true\n",
        "filter=\"has_issn:true\" # @param {type: \"string\"}\n",
        "\n",
        "response = sources_api.get_sources(\n",
        "\tfilter=filter\n",
        ")\n",
        "\n",
        "df = pd.DataFrame(response.results)\n",
        "display(df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "numeric_df = df[['id', 'display_name'] +\n",
        "\t[col for col in df.columns if df[col].dtype in ['int64', 'float64'] and col != 'relevance_score']]\n",
        "display(numeric_df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "try:\n",
        "\tprompt = \"Visualize this data\" # @param {type:\"string\"}\n",
        "\tSmartDataframe(\n",
        "\t\tnumeric_df,\n",
        "\t\tconfig={\"llm\": (OpenAI(api_token=openapi_token))},\n",
        "\t).chat(prompt)\n",
        "except:\n",
        "\tif not openapi_token:\n",
        "\t\tprint(\"Error: openapi_token not set\")\n",
        "\telse:\n",
        "\t\tprint(\"Error when creating SmartDataframe\")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "{% hint style=\"info\" %}\n",
        "It's best to [read about filters](./../../how-to-use-the-api/get-lists-of-entities/filter-entity-lists.ipynb) before trying these out. It will show you how to combine filters and build an AND, OR, or negation query\n",
        "{% endhint %}\n",
        "\n",
        "### `/sources` attribute filters\n",
        "\n",
        "You can filter using these attributes of the `Source` entity object (click each one to view their documentation on the [`Source`](./source-object.ipynb) object page):\n",
        "\n",
        "* [`apc_prices.currency`](./source-object.md#apc_prices)\n",
        "* [`apc_prices.price`](./source-object.md#apc_prices)\n",
        "* [`apc_usd`](./source-object.md#apc_usd)\n",
        "* [`cited_by_count`](./source-object.md#cited_by_count)\n",
        "* [`country_code`](./source-object.md#country_code)\n",
        "* [`host_organization`](./source-object.md#host_organization) (alias: `host_organization.id`)\n",
        "* [`host_organization_lineage`](./source-object.md#host_organization_lineage) — Use this with a publisher ID to find works from that publisher and all of its children.\n",
        "* [`ids.openalex`](./source-object.md#ids) (alias: `openalex`)\n",
        "* [`is_in_doaj`](./source-object.md#is_in_doaj)\n",
        "* [`is_oa`](./source-object.md#is_oa)\n",
        "* [`issn`](./source-object.md#issn)\n",
        "* [`publisher`](./source-object.md#publisher) — Requires exact Match. Use the [`host_organization_lineage`](./source-object.md#host_organization_lineage) filter instead if you want to find works from a publisher and all of its children.\n",
        "* [`summary_stats.2yr_mean_citedness`](./source-object.md#summary_stats) (accepts float, null, !null, can use range queries such as < >)\n",
        "* [`summary_stats.h_index`](./source-object.md#summary_stats) (accepts integer, null, !null, can use range queries)\n",
        "* [`summary_stats.i10_index`](./source-object.md#summary_stats) (accepts integer, null, !null, can use range queries)\n",
        "* [`type`](./source-object.md#type)\n",
        "* [`works_count`](./source-object.md#works_count)\n",
        "* [`x_concepts.id`](./source-object.md#x_concepts) (alias: `concepts.id` or `concept.id`)\n",
        "\n",
        "{% hint style=\"info\" %}\n",
        "Want to filter by `host_organization.display_name`? This is a two-step process:\n",
        "\n",
        "1. Find the host organization's ID by searching by `display_name` in Publishers or Institutions, depending on which type you are looking for.\n",
        "2. Filter works by `host_organization.id`.\n",
        "\n",
        "To learn more about why we do it this way, [see here.](./../works/search-works.md#why-cant-i-search-by-name-of-related-entity-author-name-institution-name-etc.)\n",
        "{% endhint %}\n",
        "\n",
        "### `/sources` convenience filters\n",
        "\n",
        "These filters aren't attributes of the [`Source`](./source-object.ipynb) object, but they're included to address some common use cases:\n",
        "\n",
        "#### `continent`\n",
        "\n",
        "Value: a String with a valid [continent filter](./../geo/continents.md#filter-by-continent)\n",
        "\n",
        "Returns: sources that are associated with the chosen continent.\n",
        "\n",
        "* Get sources that are associated with Asia\n",
        "  [`https://api.openalex.org/sources?filter=continent:asia`](https://api.openalex.org/sources?filter=continent:asia)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title { run: \"auto\", vertical-output: false }\n",
        "# https://api.openalex.org/sources?filter=continent:asia\n",
        "filter=\"continent:asia\" # @param {type: \"string\"}\n",
        "\n",
        "response = sources_api.get_sources(\n",
        "\tfilter=filter\n",
        ")\n",
        "\n",
        "df = pd.DataFrame(response.results)\n",
        "display(df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "numeric_df = df[['id', 'display_name'] +\n",
        "\t[col for col in df.columns if df[col].dtype in ['int64', 'float64'] and col != 'relevance_score']]\n",
        "display(numeric_df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "try:\n",
        "\tprompt = \"Visualize this data\" # @param {type:\"string\"}\n",
        "\tSmartDataframe(\n",
        "\t\tnumeric_df,\n",
        "\t\tconfig={\"llm\": (OpenAI(api_token=openapi_token))},\n",
        "\t).chat(prompt)\n",
        "except:\n",
        "\tif not openapi_token:\n",
        "\t\tprint(\"Error: openapi_token not set\")\n",
        "\telse:\n",
        "\t\tprint(\"Error when creating SmartDataframe\")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### `default.search`\n",
        "\n",
        "Value: a search string\n",
        "\n",
        "This works the same as using the [`search` parameter](./search-sources.md#search-sources) for Sources.\n",
        "\n",
        "#### `display_name.search`\n",
        "\n",
        "Value: a search string\n",
        "\n",
        "Returns: sources with a [`display_name`](./source-object.md#display_name) containing the given string; see the [search page](./search-sources.ipynb) for details.\n",
        "\n",
        "* Get sources with names containing \"Neurology\":\n",
        "  [`https://api.openalex.org/sources?filter=display_name.search:Neurology`](https://api.openalex.org/sources?filter=display_name.search:Neurology)``"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title { run: \"auto\", vertical-output: false }\n",
        "# https://api.openalex.org/sources?filter=display_name.search:Neurology\n",
        "filter=\"display_name.search:Neurology\" # @param {type: \"string\"}\n",
        "\n",
        "response = sources_api.get_sources(\n",
        "\tfilter=filter\n",
        ")\n",
        "\n",
        "df = pd.DataFrame(response.results)\n",
        "display(df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "numeric_df = df[['id', 'display_name'] +\n",
        "\t[col for col in df.columns if df[col].dtype in ['int64', 'float64'] and col != 'relevance_score']]\n",
        "display(numeric_df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "try:\n",
        "\tprompt = \"Visualize this data\" # @param {type:\"string\"}\n",
        "\tSmartDataframe(\n",
        "\t\tnumeric_df,\n",
        "\t\tconfig={\"llm\": (OpenAI(api_token=openapi_token))},\n",
        "\t).chat(prompt)\n",
        "except:\n",
        "\tif not openapi_token:\n",
        "\t\tprint(\"Error: openapi_token not set\")\n",
        "\telse:\n",
        "\t\tprint(\"Error when creating SmartDataframe\")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "{% hint style=\"info\" %}\n",
        "In most cases, you should use the [`search`](./search-sources.md#sources-full-search) parameter instead of this filter because it uses a better search algorithm.\n",
        "{% endhint %}\n",
        "\n",
        "#### `has_issn`\n",
        "\n",
        "Value: a Boolean (`true` or `false`)\n",
        "\n",
        "Returns: sources that have or lack an [ISSN](./source-object.md#issn), depending on the given value.\n",
        "\n",
        "* Get sources without ISSNs:\n",
        "  [`https://api.openalex.org/sources?filter=has_issn:false`](https://api.openalex.org/sources?filter=has_issn:false)``"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title { run: \"auto\", vertical-output: false }\n",
        "# https://api.openalex.org/sources?filter=has_issn:false\n",
        "filter=\"has_issn:false\" # @param {type: \"string\"}\n",
        "\n",
        "response = sources_api.get_sources(\n",
        "\tfilter=filter\n",
        ")\n",
        "\n",
        "df = pd.DataFrame(response.results)\n",
        "display(df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "numeric_df = df[['id', 'display_name'] +\n",
        "\t[col for col in df.columns if df[col].dtype in ['int64', 'float64'] and col != 'relevance_score']]\n",
        "display(numeric_df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "try:\n",
        "\tprompt = \"Visualize this data\" # @param {type:\"string\"}\n",
        "\tSmartDataframe(\n",
        "\t\tnumeric_df,\n",
        "\t\tconfig={\"llm\": (OpenAI(api_token=openapi_token))},\n",
        "\t).chat(prompt)\n",
        "except:\n",
        "\tif not openapi_token:\n",
        "\t\tprint(\"Error: openapi_token not set\")\n",
        "\telse:\n",
        "\t\tprint(\"Error when creating SmartDataframe\")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### `is_global_south`\n",
        "\n",
        "Value: a Boolean (`true` or `false`)\n",
        "\n",
        "Returns: sources that are associated with the [Global South](./../geo/regions.md#global-south).\n",
        "\n",
        "* Get sources that are located in the Global South\n",
        "  [`https://api.openalex.org/sources?filter=is_global_south:true`](https://api.openalex.org/sources?filter=is_global_south:true)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title { run: \"auto\", vertical-output: false }\n",
        "# https://api.openalex.org/sources?filter=is_global_south:true\n",
        "filter=\"is_global_south:true\" # @param {type: \"string\"}\n",
        "\n",
        "response = sources_api.get_sources(\n",
        "\tfilter=filter\n",
        ")\n",
        "\n",
        "df = pd.DataFrame(response.results)\n",
        "display(df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "numeric_df = df[['id', 'display_name'] +\n",
        "\t[col for col in df.columns if df[col].dtype in ['int64', 'float64'] and col != 'relevance_score']]\n",
        "display(numeric_df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "try:\n",
        "\tprompt = \"Visualize this data\" # @param {type:\"string\"}\n",
        "\tSmartDataframe(\n",
        "\t\tnumeric_df,\n",
        "\t\tconfig={\"llm\": (OpenAI(api_token=openapi_token))},\n",
        "\t).chat(prompt)\n",
        "except:\n",
        "\tif not openapi_token:\n",
        "\t\tprint(\"Error: openapi_token not set\")\n",
        "\telse:\n",
        "\t\tprint(\"Error when creating SmartDataframe\")"
      ],
      "outputs": [],
      "execution_count": null
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}