Embarking on a journey to bridge the analytical divide between the R and Python communities, I launched the “TidyTuesday for Python” project. This endeavor curates TidyTuesday datasets from 2023 to 2024 into a format that’s more palatable for Python aficionados. TidyTuesday, a cherished tradition within the R community, releases weekly datasets aimed at honing data visualization and wrangling skills. However, the Python community often finds these resources out of reach. By converting these datasets into Python-friendly formats (JSON and CSV), I aimed to create a common ground for shared learning and open-source collaboration.
JSON Nested Data of TidyTuesday Projects in 2023-2024
Homepage: 🤗 HuggingFace Dataset | 🏠 GitHub Source Code
Source Code: Data Scraping & Processing Code | Example Use Cases
Note: As a pilot project, the datasets from Tidy Tuesday will be added retrospectively. The current dataset incorporates all posts starting from Jan 2023. Depending on time and availability, data dating back further will be considered.
Uses
- This dataset can be used for data visualization, statistical analysis, and machine learning NLP tasks. Since it’s a compilation of diverse datasets, it supports a variety of tasks and can be used to benchmark different data processing and visualization techniques in Python.
- This dataset can also be your starting point in discovering various topics of exploration, stepping your feet into unknown territories, and offering insights into the impact that data can realize. Direct view and download links are provided within the data structure.
- By referring to TidyTuesday, you have the ability to discover the specialties of both Python and R, choose wisely on different packages or features that each language offers, or even learn a new language based on the one you’re familiar with!
Dataset Structure
The dataset is stored in both JSON nested hierarchical structure and CSV format.
- Full set:
tidytuesday_json.jsonandtidytuesday_2023_2024.csv - Train set:
tidytuesday_json_train.jsonandtidytuesday_train.csv - Validation set:
tidytuesday_json_val.jsonandtidytuesday_val.csv
Data Instances
{
"date_posted": "YYYY-MM-DD",
"project_name": "Example project name for one post",
"project_source": ["https://example-tidytuesday-post-source/", ...],
"description": "Project description excerpted from TidyTuesday repo",
"data_source_url": "https://example-tidytuesday/date/repo",
"data_dictionary": [
{
"variable": [variable1, variable2, ...],
"class": [class_type1, class_type2, ...],
"description": ["Description of var1", "Description of var2", ...]
},
...
],
"data": {
"file_name": [
"data1.csv",
"data2.csv",
...
],
"file_url": [
"https://example-tidytuesday/view-link-to-data1",
"https://example-tidytuesday/view-link-to-data2",
...
]
},
"data_load": {
"file_name": [
"data1.csv",
"data2.csv",
...
],
"file_url": [
"https://example-tidytuesday/download-link-to-data1",
"https://example-tidytuesday/download-link-to-data2",
...
]
}
}
Data Fields
| Key / Variable | Type | Description |
|---|---|---|
date_posted |
str | The date when the weekly project was posted (YYYY-MM-DD) |
project_name |
str | The name of the weekly project |
project_source |
List[str] | A list of URLs to the project’s data and information source |
description |
str | A brief excerpt of the project and dataset description |
data_source_url |
str | The URL to the weekly project repository |
data_dictionary |
List[Dict[str: List[str]]] | A list of dictionaries, each containing the variable names, classes, and descriptions for each dataset |
data |
Dict[str: List[str]] | A dictionary of dataset names and links to view. The values of the keys file_name and file_url are organized in list fashion |
data_load |
Dict[str: List[str]] | A dictionary of dataset names and links to direct download. The values of the keys file_name and file_url are organized in list fashion |
Dataset Creation
Curation Rationale
The genesis of this dataset was rooted in a personal quest to bridge the gap between the R and Python communities, particularly in the realm of data visualization and wrangling. As an avid data enthusiast, I often encountered challenges in discovering captivating projects and datasets to fuel my own analytical endeavors. Tidy Tuesday, with its weekly release of diverse and engaging datasets, presented a goldmine of opportunities. However, I noticed a disconnect - these datasets, predominantly tailored for the R community, were not as readily accessible or structured for Python users. This realization sparked the ambition to create a centralized, Python-friendly repository of TidyTuesday’s 2023 datasets. By doing so, the dataset aims to democratize access to these rich resources, fostering a shared learning environment and stimulating innovation across both R and Python communities.
The dataset has been crafted to include essential metadata like the date posted, project name, data source, and a comprehensive description. What makes it stand out is the inclusion of a nested data structure for both the data dictionary and the actual data, tailored to enhance user experience and accessibility. This structure not only addresses the challenges I faced in finding and utilizing interesting datasets for personal projects but also streamlines the process for others facing similar hurdles. The ultimate goal is to create a vibrant, interactive playground where enthusiasts from both communities can converge, collaborate, and contribute, thereby enriching the data science landscape with diverse perspectives and insights.
Source Data
The major source of data is directly from the TidyTuesday official site. Initial data collection and normalization were conducted collaboratively by the organizers of Tidy Tuesday and data runners within the R community. Since it is an R-oriented initiative, the source language, example files, and authors’ dataset cleanings were mostly done in R.
Who are the original source data producers?
The source data come from a variety of fields, including but not limited to the R community, TidyTuesday collaborators, interesting web postings, and scholarly papers. Details of each week’s data sources can be retrieved from the project_source variable. Descriptions of the sources can also be found as part of the description variable.
Data Collection and Processing
The data collection process involved several key stages, starting with the use of Python libraries requests and BeautifulSoup to programmatically navigate the repository and extract post dates and URLs for selected years. This initial step was crucial for mapping out the structure of the data collection effort.
Subsequent to this foundational step, detailed scraping scripts were developed to delve into individual post URLs, extracting the variable-specific information. Special attention was paid to the HTML structure of the TidyTuesday posts to ensure accurate and efficient data extraction. This process was iterative and adaptive, with scripts being refined to handle inconsistencies in post structures and to ensure comprehensive data capture.
The resultant dataset is presented in both CSV and JSON formats, providing flexibility for analysis and application. The JSON format, in particular, was structured to facilitate easy integration into Python environments, with nested objects and hierarchical data representations designed for intuitive access and manipulation. This meticulous approach to data collection and processing ensures that the dataset is not only comprehensive and detailed but also ready for immediate use in a wide range of data analysis, machine learning, and data visualization projects. The full dataset, along with subsets for training and validation, has been made available on GitHub and published on HuggingFace, ensuring open access for researchers, data scientists, and enthusiasts interested in exploring social data trends. My scraping and preprocessing nodebook is also available publicly!